Bank Churn Prediction

Problem Statement

Context

Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.

Objective

You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.

Data Dictionary

• CustomerId: Unique ID which is assigned to each customer

• Surname: Last name of the customer

• CreditScore: It defines the credit history of the customer.

• Geography: A customer’s location

• Gender: It defines the Gender of the customer

• Age: Age of the customer

• Tenure: Number of years for which the customer has been with the bank

• NumOfProducts: refers to the number of products that a customer has purchased through the bank.

• Balance: Account balance

• HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.

• EstimatedSalary: Estimated salary

• isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )

• Exited : whether or not the customer left the bank within six month. It can take two values

** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )

# Installing the libraries with the specified version.
#!pip3 install tensorflow==2.15.0 scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==2.0.3 imbalanced-learn==0.10.1 -q --user
#!pip3 install tensorflow==2.16.1

Importing necessary libraries

# Libraries to help with reading and manipulating data
import pandas as pd
import numpy as np

# libaries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Library to split data
from sklearn.model_selection import train_test_split

# library to import to standardize the data
from sklearn.preprocessing import StandardScaler, RobustScaler, LabelEncoder

# importing different functions to build models
import tensorflow as tf
from tensorflow import keras
from keras import backend
from keras.models import Sequential
from keras.layers import Dense, Dropout, BatchNormalization

# importing SMOTE
from imblearn.over_sampling import SMOTE

# importing metrics
from sklearn.metrics import confusion_matrix,roc_curve,classification_report,recall_score

import random

# Library to avoid the warnings
import warnings
warnings.filterwarnings("ignore")

2024-06-05 19:07:29.059233: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

Loading the dataset

#load the dataset
churn = pd.read_csv("churn.csv")

Data Overview

# view the first 5 rows of the data
churn.head()

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	2	15647311	Hill	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	3	15619304	Onio	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	4	15701354	Boni	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

# view the last 5 rows of the data
churn.tail()

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
9995	9996	15606229	Obijiaku	771	France	Male	39	5	0.00	2	1	0	96270.64	0
9996	9997	15569892	Johnstone	516	France	Male	35	10	57369.61	1	1	1	101699.77	0
9997	9998	15584532	Liu	709	France	Female	36	7	0.00	1	0	1	42085.58	1
9998	9999	15682355	Sabbatini	772	Germany	Male	42	3	75075.31	2	1	0	92888.52	1
9999	10000	15628319	Walker	792	France	Female	28	4	130142.79	1	1	0	38190.78	0

• Age, Balance, Tenure, CreditScore, and EstimatedSalary will probably have to be normalized.
• Geography and Gender will have to be one hot encoded to be of use in our model.

churn.shape

(10000, 14)

10000 rows and 14 columns

#Check the data types of the columns for the dataset
churn.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   RowNumber        10000 non-null  int64  
 1   CustomerId       10000 non-null  int64  
 2   Surname          10000 non-null  object 
 3   CreditScore      10000 non-null  int64  
 4   Geography        10000 non-null  object 
 5   Gender           10000 non-null  object 
 6   Age              10000 non-null  int64  
 7   Tenure           10000 non-null  int64  
 8   Balance          10000 non-null  float64
 9   NumOfProducts    10000 non-null  int64  
 10  HasCrCard        10000 non-null  int64  
 11  IsActiveMember   10000 non-null  int64  
 12  EstimatedSalary  10000 non-null  float64
 13  Exited           10000 non-null  int64  
dtypes: float64(2), int64(9), object(3)
memory usage: 1.1+ MB

#Checking the Statistical Summary
churn.describe().T

	count	mean	std	min	25%	50%	75%	max
RowNumber	10000.0	5.000500e+03	2886.895680	1.00	2500.75	5.000500e+03	7.500250e+03	10000.00
CustomerId	10000.0	1.569094e+07	71936.186123	15565701.00	15628528.25	1.569074e+07	1.575323e+07	15815690.00
CreditScore	10000.0	6.505288e+02	96.653299	350.00	584.00	6.520000e+02	7.180000e+02	850.00
Age	10000.0	3.892180e+01	10.487806	18.00	32.00	3.700000e+01	4.400000e+01	92.00
Tenure	10000.0	5.012800e+00	2.892174	0.00	3.00	5.000000e+00	7.000000e+00	10.00
Balance	10000.0	7.648589e+04	62397.405202	0.00	0.00	9.719854e+04	1.276442e+05	250898.09
NumOfProducts	10000.0	1.530200e+00	0.581654	1.00	1.00	1.000000e+00	2.000000e+00	4.00
HasCrCard	10000.0	7.055000e-01	0.455840	0.00	0.00	1.000000e+00	1.000000e+00	1.00
IsActiveMember	10000.0	5.151000e-01	0.499797	0.00	0.00	1.000000e+00	1.000000e+00	1.00
EstimatedSalary	10000.0	1.000902e+05	57510.492818	11.58	51002.11	1.001939e+05	1.493882e+05	199992.48
Exited	10000.0	2.037000e-01	0.402769	0.00	0.00	0.000000e+00	0.000000e+00	1.00

# check for missing values in the data, express as %
round(churn.isnull().sum() / churn.isnull().count() * 100, 2)

RowNumber          0.0
CustomerId         0.0
Surname            0.0
CreditScore        0.0
Geography          0.0
Gender             0.0
Age                0.0
Tenure             0.0
Balance            0.0
NumOfProducts      0.0
HasCrCard          0.0
IsActiveMember     0.0
EstimatedSalary    0.0
Exited             0.0
dtype: float64

• No missing values.

# check for duplicate values
churn.duplicated().sum()

0

• No duplicate values.

#Checking for unique values for each of the column
churn.nunique()

RowNumber          10000
CustomerId         10000
Surname             2932
CreditScore          460
Geography              3
Gender                 2
Age                   70
Tenure                11
Balance             6382
NumOfProducts          4
HasCrCard              2
IsActiveMember         2
EstimatedSalary     9999
Exited                 2
dtype: int64

• Age has only 70 unique values, likely meaning large groupings of similar ages.
• Geography and Gender look to be categorical in nature.
• We have many continuous variables like CreditScore, Balance, EstimatedSalary, etc.
• RowNumber consists of uniques ID for each row and will not add value to the modeling
• CustomerId consists of uniques ID for customers and will not add value to the modeling
• Surname consists of last names customers and will not add value to this modeling

# Can see that several of these columns have limited data types, so would be better as categories
# Converting the data type of columns with category like data to 'category'

# don't convert Exited, since we will remove later and its the target var.
category_col = ['Geography','Gender']

churn[category_col] = churn[category_col].astype('category')

# take a look at the data types after conversion
churn.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype   
---  ------           --------------  -----   
 0   RowNumber        10000 non-null  int64   
 1   CustomerId       10000 non-null  int64   
 2   Surname          10000 non-null  object  
 3   CreditScore      10000 non-null  int64   
 4   Geography        10000 non-null  category
 5   Gender           10000 non-null  category
 6   Age              10000 non-null  int64   
 7   Tenure           10000 non-null  int64   
 8   Balance          10000 non-null  float64 
 9   NumOfProducts    10000 non-null  int64   
 10  HasCrCard        10000 non-null  int64   
 11  IsActiveMember   10000 non-null  int64   
 12  EstimatedSalary  10000 non-null  float64 
 13  Exited           10000 non-null  int64   
dtypes: category(2), float64(2), int64(9), object(1)
memory usage: 957.4+ KB

# view the statistical summary of the non-numerical columns in the data
churn.describe(exclude=np.number).T

	count	unique	top	freq
Surname	10000	2932	Smith	32
Geography	10000	3	France	5014
Gender	10000	2	Male	5457

var_col = [
    "CreditScore",
    "Geography",
    "Gender",
    "Age",
    "Tenure",
    "Balance",
    "NumOfProducts",
    "HasCrCard",
    "IsActiveMember",
    "EstimatedSalary",
    "Exited"
]

i=0
#summarize all values of categories
for column in var_col:
    print(var_col[i])
    print("-"*55)
    print(churn[column].value_counts())
    print("\n")
    i+=1
    

CreditScore
-------------------------------------------------------
CreditScore
850    233
678     63
655     54
705     53
667     53
      ... 
351      1
365      1
382      1
373      1
419      1
Name: count, Length: 460, dtype: int64


Geography
-------------------------------------------------------
Geography
France     5014
Germany    2509
Spain      2477
Name: count, dtype: int64


Gender
-------------------------------------------------------
Gender
Male      5457
Female    4543
Name: count, dtype: int64


Age
-------------------------------------------------------
Age
37    478
38    477
35    474
36    456
34    447
     ... 
84      2
88      1
82      1
85      1
83      1
Name: count, Length: 70, dtype: int64


Tenure
-------------------------------------------------------
Tenure
2     1048
1     1035
7     1028
8     1025
5     1012
3     1009
4      989
9      984
6      967
10     490
0      413
Name: count, dtype: int64


Balance
-------------------------------------------------------
Balance
0.00         3617
105473.74       2
130170.82       2
130142.79       1
117419.35       1
             ... 
88381.21        1
155060.41       1
57369.61        1
75075.31        1
150725.53       1
Name: count, Length: 6382, dtype: int64


NumOfProducts
-------------------------------------------------------
NumOfProducts
1    5084
2    4590
3     266
4      60
Name: count, dtype: int64


HasCrCard
-------------------------------------------------------
HasCrCard
1    7055
0    2945
Name: count, dtype: int64


IsActiveMember
-------------------------------------------------------
IsActiveMember
1    5151
0    4849
Name: count, dtype: int64


EstimatedSalary
-------------------------------------------------------
EstimatedSalary
24924.92     2
196526.55    1
158590.25    1
118913.53    1
38190.78     1
            ..
136869.31    1
113308.29    1
159334.93    1
20393.44     1
6847.73      1
Name: count, Length: 9999, dtype: int64


Exited
-------------------------------------------------------
Exited
0    7963
1    2037
Name: count, dtype: int64

churn_drop = churn.copy()

# RowNumber why?, CustomerId consists of uniques ID for customer, Surname may not be unique but not needed and will not add value to the modeling so delete.
churn_drop.drop(['RowNumber', 'CustomerId', 'Surname'], axis=1, inplace=True)

# view the first 5 rows of the data
churn_drop.head()

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Exploratory Data Analysis

# Number each graph/figure start with 1
figNo=1

#Make it easier to see data by update to Yes/No and Categorical for visualization
churn_visual = churn_drop.copy()

churn_visual = churn_visual.replace({'Exited':{0:'No', 1:'Yes'},
                                     'HasCrCard':{0:'No',1:'Yes'},
                                     'IsActiveMember':{0:'No',1:'Yes'}}) #just these for viz

Univariate Analysis

# function to plot a boxplot and a histogram along the same scale.


def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to show the density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a star will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram

# function to create labeled barplots


def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot

Observations on CreditScore

Title = 'CreditScore'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

histogram_boxplot(churn_visual, Title)

Fig 1 - CreditScore

Observations

• CreditScore distribution is left skewed.
• The plot shows outliers below 400.
• The Median is about 660 and the mean is very close to the median
• There are a large number of customers with CreditScore above 840.
• These values appear valid due to wide variations in credit scores and thus will not be adjusted.

Observations on Age

Title = 'Age'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

histogram_boxplot(churn_visual, Title)

Fig 2 - Age

Observations

• The distribution of Age is skewed to the right.
• The plot shows outliers at the right side (high end of age) above 60.
• The median is about 37 and the mean is about 39.
• We will not adjust these outliers as they represent real ages and trends in age.

Observations on Balance

Title = 'Balance'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

histogram_boxplot(churn_visual, Title) 

Fig 3 - Balance

Observations

• The distribution of Balance is would be normal if not for the huge number of customers with 0 Balance (3617).
• This may indicate customers with a credit card only and not a bank account.
• No outliers.

Observations on Estimated Salary

Title = 'EstimatedSalary'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

histogram_boxplot(churn_visual, Title)

Fig 4 - EstimatedSalary

Observations

• The distribution of EstimatedSalary is close to even across the range.
• The plot shows no outliers.
• This variable will likely have little influence.

Observations on Exited

Title = 'Exited'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 5 - Exited

Observations

• Almost 80% of customers churned.
• The data set is imbalanced.

Observations on Geography

Title = 'Geography'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True) 

Fig 6 - Geography

Observations

• 50% of customers are in France.
• 25% are in Germany and 25% are in Spain.

Observations on Gender

Title = 'Gender'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 7 - Gender

Observations

• Almost 55% of customers are male.

Observations on Tenure

Title = 'Tenure'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 8 - Tenure

Observations

• The distribution of Tenure is fairly even between 1-9 years.
• Similarly, customers at the low end (0) and high end (10) are about the same distribution.
• This data is strange in that it appears there hasn't been much growth this year in customers, and no one over 10 years.

Observations on Number of Products

Title = 'NumOfProducts'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 9 - NumOfProducts

Observations

• Around 51% of all customers have only 1 product, while ~46% have 2.
• Those with more than 2 are a very small minority.

Observations on Has Credit Card

Title = 'HasCrCard'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 10 - HasCrCard

Observations

• 70.5% of customers have a credit card.

Observations on Is Active Member

Title = 'IsActiveMember'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

labeled_barplot(churn_visual, Title, perc=True)

Fig 11 - IsActiveMember

Observations

• Almost 52% of customers are active.

Bivariate Analysis

# function to plot stacked bar chart


def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left",
        frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()# function to plot stacked bar chart

print("Fig "+str(figNo)+" - "+ "Pairplot")
figNo+=1


churn_pair = churn.copy()
churn_pair = churn_pair.replace({'Exited':{0:'No', 1:'Yes'}}) # do this so all data viz can benefit

sns.pairplot(churn_pair, diag_kind="kde",kind="scatter", hue="Exited", dropna=True, corner=True)
plt.show()

Fig 12 - Pairplot

Observations

• CreditScore for churn/no churn have a similar distribution.
• Age shows that middle-aged customers are more likely to churn than younger or older customers.
• Tenure shows similar churn/no churn distribution.
• Balance shows that smaller balances churn less than larger balances.
• NumOfProducts shows that less products churn more than more products.
• HasCrCard shows similar distributions for churn/no churn.
• IsActiveMember shows that Active members churn less than non active members.
• EstimatedSalary shows similar distributions for churn/no churn customers.

Correlation plot

Title = 'Correlation Plot'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

plt.figure(figsize=(15, 7))
sns.heatmap(churn_visual.corr(numeric_only=True), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()

Fig 13 - Correlation Plot

Observations

• None of these items are strongly correlated.
• Balance and NumOfProducts is slighly negatively correlated.
• Not much insight to be gained here.

### Function to plot distributions

def distribution_plot_wrt_target(data, predictor, target):

    fig, axs = plt.subplots(2, 2, figsize=(12, 10))

    target_uniq = data[target].unique()

    axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
    sns.histplot(
        data=data[data[target] == target_uniq[0]],
        x=predictor,
        kde=True,
        ax=axs[0, 0],
        color="teal",
    )

    axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
    sns.histplot(
        data=data[data[target] == target_uniq[1]],
        x=predictor,
        kde=True,
        ax=axs[0, 1],
        color="orange",
    )

    axs[1, 0].set_title("Boxplot w.r.t target")
    sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")

    axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
    sns.boxplot(
        data=data,
        x=target,
        y=predictor,
        ax=axs[1, 1],
        showfliers=False,
        palette="gist_rainbow",
    )

    plt.tight_layout()
    plt.show()

Exited Vs Geography

Title = 'Exited Vs Geography'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

stacked_barplot(churn_visual, "Geography", "Exited" )

Fig 14 - Exited Vs Geography
Exited       No   Yes    All
Geography                   
All        7963  2037  10000
Germany    1695   814   2509
France     4204   810   5014
Spain      2064   413   2477
------------------------------------------------------------------------------------------------------------------------

Observations

• Spanish customers churn similarly in proportion to French customers
• German customers churn at a higher ration, but also in overall number.

Exited Vs Gender

Title = 'Exited Vs Gender'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

stacked_barplot(churn_visual, "Gender", "Exited")

Fig 15 - Exited Vs Gender
Exited    No   Yes    All
Gender                   
All     7963  2037  10000
Female  3404  1139   4543
Male    4559   898   5457
------------------------------------------------------------------------------------------------------------------------

Observations

• Females churn at a higher proportion than males and represent more of the overall churned customers.

Exited Vs Has Credit Card

Title = 'Exited Vs Has Credit Card'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

stacked_barplot(churn_visual, "HasCrCard", "Exited")

Fig 16 - Exited Vs Has Credit Card
Exited       No   Yes    All
HasCrCard                   
All        7963  2037  10000
Yes        5631  1424   7055
No         2332   613   2945
------------------------------------------------------------------------------------------------------------------------

Observations

• Customers with or without credit cards churn at about the same proportion.
• Customers with a credit card make up the bulk of churners.

Exited Vs Is active member

Title = 'Exited Vs Is active member'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

stacked_barplot(churn_visual, "IsActiveMember", "Exited")

Fig 17 - Exited Vs Is active member
Exited            No   Yes    All
IsActiveMember                   
All             7963  2037  10000
No              3547  1302   4849
Yes             4416   735   5151
------------------------------------------------------------------------------------------------------------------------

Observations

• Active customers churn less proportionally.
• Non-active customers make up the bulk of churners.

Exited Vs Credit Score

Title = 'Exited Vs Credit Score'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "CreditScore", "Exited")

Fig 18 - Exited Vs Credit Score

Observations

• The distribution of CreditScore is is similar for both groups of customers.
• There are several outliers with CreditScore below 400 that have churned.
• We will not adjust these outliers as they represent real trends.

Exited Vs Age

Title = 'Exited Vs Age'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "Age", "Exited")

Fig 19 - Exited Vs Age

Observations

• The plot shows many outliers at the higher end of age
• Customers between 40-50 show a tendency to churn.
• Customers between 30-40 and above 60 show low tendency to churn.
• We will not adjust these outliers as they represent real ages and trends in age.

Exited Vs Tenure

Title = 'Exited Vs Tenure'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "Tenure", "Exited")

Fig 20 - Exited Vs Tenure

Observations

• No outliers.
• Distribution is similar in years 1-9, and very low in 0+10

Exited Vs Balance

Title = 'Exited Vs Balance'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "Balance", "Exited")

Fig 21 - Exited Vs Balance

Observations

• Customers with a lower Balance tend to churn more.

Exited Vs Number of Products

Title = 'Exited Vs Number of Products'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "NumOfProducts", "Exited")

Fig 22 - Exited Vs Number of Products

Observations

• Customers with 1 product churn the most. Since they only have 1 product, they probably have little incentive not to churn.
• Customers with 2 products churn the least.

Exited Vs Estimated Salary

Title = 'Exited Vs Estimated Salary'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1

distribution_plot_wrt_target(churn_visual, "EstimatedSalary", "Exited")

Fig 23 - Exited Vs Estimated Salary

Observations

• The distribution of EstimatedSalary is similar across both groups.
• This variable will likely not influence much in the model.

Data Preprocessing

#Copy churn_data
churn_data = churn_drop.copy()

Dummy Variable Creation

# Creating dummy variables for categorical variables
# churn_data = pd.get_dummies(data=churn_data, drop_first=True)

dummies = pd.get_dummies(churn_data[['Geography', 'Gender']], drop_first=True)
churn_data = pd.concat([churn_data.drop(['Geography', 'Gender'],axis=1), dummies],axis=1)

churn_data.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
0	619	42	2	0.00	1	1	1	101348.88	1	False	False	False
1	608	41	1	83807.86	1	0	1	112542.58	0	False	True	False
2	502	42	8	159660.80	3	1	0	113931.57	1	False	False	False
3	699	39	1	0.00	2	0	0	93826.63	0	False	False	False
4	850	43	2	125510.82	1	1	1	79084.10	0	False	True	False

churn_data.shape

(10000, 12)

Data Normalization

churn_data_scaled=churn_data.copy()

churn_data_scaled.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 12 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   CreditScore        10000 non-null  int64  
 1   Age                10000 non-null  int64  
 2   Tenure             10000 non-null  int64  
 3   Balance            10000 non-null  float64
 4   NumOfProducts      10000 non-null  int64  
 5   HasCrCard          10000 non-null  int64  
 6   IsActiveMember     10000 non-null  int64  
 7   EstimatedSalary    10000 non-null  float64
 8   Exited             10000 non-null  int64  
 9   Geography_Germany  10000 non-null  bool   
 10  Geography_Spain    10000 non-null  bool   
 11  Gender_Male        10000 non-null  bool   
dtypes: bool(3), float64(2), int64(7)
memory usage: 732.6 KB

# defining the list of numerical columns
cols_list = ["CreditScore","Age","Balance","EstimatedSalary","NumOfProducts","Tenure"]

# creating an instance of the RobustScaler since it will work better with outliers than standard
scaler = RobustScaler()

#X_train[cols_list] = sc.fit_transform(X_train[cols_list])
churn_data_scaled[cols_list] = scaler.fit_transform(churn_data_scaled[cols_list])

#Printing the maximum value and the minimum value of the independent variable.
print(churn_data_scaled.max())
print(churn_data_scaled.min())

CreditScore          1.477612
Age                  4.583333
Tenure                   1.25
Balance              1.204124
NumOfProducts             3.0
HasCrCard                   1
IsActiveMember              1
EstimatedSalary      1.014356
Exited                      1
Geography_Germany        True
Geography_Spain          True
Gender_Male              True
dtype: object
CreditScore         -2.253731
Age                 -1.583333
Tenure                  -1.25
Balance              -0.76148
NumOfProducts             0.0
HasCrCard                   0
IsActiveMember              0
EstimatedSalary     -1.018257
Exited                      0
Geography_Germany       False
Geography_Spain         False
Gender_Male             False
dtype: object

churn_data_scaled.shape

(10000, 12)

churn_data_scaled.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
0	-0.246269	0.416667	-0.75	-0.761480	0.0	1	1	0.011739	1	False	False	False
1	-0.328358	0.333333	-1.00	-0.104906	0.0	0	1	0.125512	0	False	True	False
2	-1.119403	0.416667	0.75	0.489346	2.0	1	0	0.139630	1	False	False	False
3	0.350746	0.166667	-1.00	-0.761480	1.0	0	0	-0.064717	0	False	False	False
4	1.477612	0.500000	-0.75	0.221806	0.0	1	1	-0.214561	0	False	True	False

Train-validation-test Split

X = churn_data_scaled.drop(['Exited'],axis=1)
y = churn_data_scaled['Exited']

# Splitting the dataset into the Training and Testing set.

X_large, X_test, y_large, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42,stratify=y,shuffle = True)

# Splitting the dataset into the Training and Validation set.

X_train, X_val, y_train, y_val = train_test_split(X_large, y_large, test_size = 0.25, random_state = 42,stratify=y_large, shuffle = True)

print(X_train.shape, X_val.shape, X_test.shape)

(6000, 11) (2000, 11) (2000, 11)

print(y_train.shape, y_val.shape, y_test.shape)

(6000,) (2000,) (2000,)

X_train.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Geography_Germany	Geography_Spain	Gender_Male
1995	-0.507463	0.583333	0.00	-0.011961	1.0	1	1	0.064755	False	False	False
2724	-1.485075	0.083333	0.75	0.183515	0.0	1	0	0.299880	True	False	False
5224	1.126866	0.500000	-0.50	-0.761480	0.0	1	0	-0.286041	False	True	True
7697	-0.380597	0.333333	-0.50	-0.761480	1.0	1	0	-0.466032	False	True	False
1226	-0.902985	0.416667	0.25	-0.069523	1.0	1	0	-0.253543	True	False	False

X_val.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Geography_Germany	Geography_Spain	Gender_Male
6263	-1.544776	0.000000	-0.50	-0.761480	1.0	1	1	0.811278	False	False	True
7644	0.171642	-0.750000	1.00	-0.761480	0.0	1	0	0.344734	False	False	True
429	-0.626866	0.250000	-1.00	0.016327	0.0	0	0	0.349714	True	False	True
647	-0.552239	0.083333	0.50	-0.117038	0.0	1	0	-0.926929	False	False	True
8353	-0.955224	-0.416667	0.25	-0.761480	0.0	1	1	0.332038	False	True	False

X_test.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Geography_Germany	Geography_Spain	Gender_Male
5702	-0.500000	-0.083333	0.50	-0.761480	1.0	1	0	-0.060078	False	False	True
3667	-0.947761	-0.333333	-0.25	0.264996	1.0	0	0	-0.458611	True	False	True
1617	-0.708955	0.250000	-0.25	-0.761480	1.0	0	1	0.053256	False	True	False
5673	-0.097015	-0.250000	0.00	0.330564	1.0	0	0	-0.673305	False	True	True
4272	-0.089552	-0.250000	-0.50	-0.151764	0.0	1	1	0.694721	False	True	False

print(y_val.max())
print(y_val.min())

1
0

print(X_train.shape[0], "train samples")
print(X_val.shape[0], "validation samples")
print(X_test.shape[0], "test samples")

6000 train samples
2000 validation samples
2000 test samples

Model Building

Model Evaluation Criterion

Our model needs to predict customer churn.

The nature of predictions made by the classification model will translate as follows:

• True positives (TP) are failures correctly predicted by the model.

• True negatives (TN) are correctly predicted by the model.

• False negatives (FN) are real failures where it is not detected by the model.

• False positives (FP) are failures detected by the model when it should not have been.

• accuracy - how many of our predictions were true

• precision - of our postive predictions, how many were true?

• recall - out of all the ones that should be true, how many did we correctly predict

• f1 - how effective we make the tradeoff between precision and recall

Since a bank wants to detect customers who are at risk of leaving in the next 6 months, they would want to focus on recall to maximize detection of customers at risk and avoid false negatives.

# defining the batch size and # epochs upfront
epochs = 50
batch_size = 64

Let's create a function for plotting the confusion matrix

def make_confusion_matrix(actual_targets, predicted_targets):
    """
    To plot the confusion_matrix with percentages

    actual_targets: actual target (dependent) variable values
    predicted_targets: predicted target (dependent) variable values
    """
    cm = confusion_matrix(actual_targets, predicted_targets)
    
    group_names = ['True Positive','False Negative','False Positive','True Negative']
    categories = [ 'Not Exiting','Exiting']
    group_labels = ["{}\n".format(value) for value in group_names]
    group_counts = ["{0:0.0f}\n".format(value) for value in cm.flatten()]
    group_percentages = ["{0:.2%}".format(value) for value in cm.flatten()/np.sum(cm)]
    
    box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
    box_labels = np.asarray(box_labels).reshape(cm.shape[0],cm.shape[1])

    accuracy  = np.trace(cm) / float(np.sum(cm))
    precision = cm[1,1] / sum(cm[:,1])
    recall    = cm[1,1] / sum(cm[1,:])
    f1_score  = 2*precision*recall / (precision + recall)

    stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
                accuracy,precision,recall,f1_score)

    plt.figure(figsize=(6, 4))
    sns.heatmap(cm, annot=box_labels,fmt="",cmap='Blues',xticklabels=categories,yticklabels=categories)
    categories = [ 'Not Exiting','Exiting']
    plt.ylabel("True label")
    plt.xlabel('Predicted label' + stats_text)

def plot_train_vs_val_loss(loss, val_loss):
    """
    Plotting Train Loss vs Validation Loss

    loss: saved loss values from each epoch
    val_loss: saved validation loss values from each epoch
    """

    plt.plot(loss)
    plt.plot(val_loss)
    plt.title('model loss')
    plt.ylabel('Loss')
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='upper left')
    plt.show()

def plot_train_vs_val_recall(recall, val_recall):
    """
    Plotting Train recall vs Validation recall

    recall: saved recall values from each epoch
    val_recall: saved validation recall values from each epoch
    """

    plt.plot(recall)
    plt.plot(val_recall)
    plt.title('model recall')
    plt.ylabel('recall')
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='upper left')
    plt.show()

# defining a function to compute different metrics to check performance of a classification model built using statsmodels
def model_performance_classification(
    model, predictors, target, threshold=0.5
):
    """
    Function to compute different metrics to check classification model performance

    model: classifier
    predictors: independent variables
    target: dependent variable
    threshold: threshold for classifying the observation as class 1
    """

    # checking which probabilities are greater than threshold
    pred = model.predict(predictors) > threshold
    # pred_temp = model.predict(predictors) > threshold
    # # rounding off the above values to get classes
    # pred = np.round(pred_temp)

    acc = accuracy_score(target, pred)  # to compute Accuracy
    recall = recall_score(target, pred, average='weighted')  # to compute Recall
    precision = precision_score(target, pred, average='weighted')  # to compute Precision
    f1 = f1_score(target, pred, average='weighted')  # to compute F1-score

    # creating a dataframe of metrics
    df_perf = pd.DataFrame(
        {"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},
        index=[0],
    )

    return df_perf

#Save metrics for later comparison, since we are optimizing for recall, we'll save recall
training_metrics = pd.DataFrame(columns=["recall"])
validation_metrics = pd.DataFrame(columns=["recall"])

Neural Network (64,32,16,8) with SGD Optimizer

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SGD"

#Initializing the neural network
sgd_model = Sequential()
sgd_model.add(Dense(64, activation='relu', input_dim = X_train.shape[1]))
sgd_model.add(Dense(32, activation='relu'))
sgd_model.add(Dense(16, activation='relu'))
sgd_model.add(Dense(8, activation='relu'))
sgd_model.add(Dense(1, activation = 'sigmoid'))

# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

check if this is the correct metric

## compile the model with binary cross entropy as loss function and recall as the metric.
sgd_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

sgd_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

check batch size and epochs

# Fitting the ANN
sgd_history = sgd_model.fit(
    X_train, y_train,
    batch_size=batch_size, 
    epochs=epochs,
    validation_data=(X_val,y_val),
    verbose=1
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6280 - recall: 0.0536 - val_loss: 0.5475 - val_recall: 0.0000e+00
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5328 - recall: 0.0000e+00 - val_loss: 0.5128 - val_recall: 0.0000e+00
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5050 - recall: 0.0000e+00 - val_loss: 0.4996 - val_recall: 0.0000e+00
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4929 - recall: 0.0000e+00 - val_loss: 0.4897 - val_recall: 0.0000e+00
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4834 - recall: 0.0000e+00 - val_loss: 0.4802 - val_recall: 0.0000e+00
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4739 - recall: 0.0000e+00 - val_loss: 0.4701 - val_recall: 0.0000e+00
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4638 - recall: 0.0000e+00 - val_loss: 0.4603 - val_recall: 0.0000e+00
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4541 - recall: 0.0000e+00 - val_loss: 0.4514 - val_recall: 0.0000e+00
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4455 - recall: 0.0000e+00 - val_loss: 0.4444 - val_recall: 0.0000e+00
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4388 - recall: 0.0000e+00 - val_loss: 0.4392 - val_recall: 0.0000e+00
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4337 - recall: 0.0000e+00 - val_loss: 0.4355 - val_recall: 0.0000e+00
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4299 - recall: 0.0000e+00 - val_loss: 0.4328 - val_recall: 0.0000e+00
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4272 - recall: 8.9250e-04 - val_loss: 0.4308 - val_recall: 0.0025
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4250 - recall: 0.0036 - val_loss: 0.4292 - val_recall: 0.0074
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4232 - recall: 0.0166 - val_loss: 0.4278 - val_recall: 0.0123
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4216 - recall: 0.0231 - val_loss: 0.4265 - val_recall: 0.0344
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4201 - recall: 0.0450 - val_loss: 0.4253 - val_recall: 0.0516
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4186 - recall: 0.0706 - val_loss: 0.4242 - val_recall: 0.0713
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4173 - recall: 0.0923 - val_loss: 0.4232 - val_recall: 0.0885
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4161 - recall: 0.0992 - val_loss: 0.4221 - val_recall: 0.1081
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4148 - recall: 0.1107 - val_loss: 0.4210 - val_recall: 0.1302
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4136 - recall: 0.1321 - val_loss: 0.4199 - val_recall: 0.1499
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4123 - recall: 0.1580 - val_loss: 0.4188 - val_recall: 0.1646
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4111 - recall: 0.1710 - val_loss: 0.4177 - val_recall: 0.1744
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4098 - recall: 0.1894 - val_loss: 0.4165 - val_recall: 0.1843
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4085 - recall: 0.2009 - val_loss: 0.4153 - val_recall: 0.1966
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4072 - recall: 0.2161 - val_loss: 0.4140 - val_recall: 0.2039
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4058 - recall: 0.2279 - val_loss: 0.4127 - val_recall: 0.2088
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4043 - recall: 0.2404 - val_loss: 0.4113 - val_recall: 0.2138
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4028 - recall: 0.2485 - val_loss: 0.4098 - val_recall: 0.2310
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4012 - recall: 0.2551 - val_loss: 0.4083 - val_recall: 0.2383
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3995 - recall: 0.2672 - val_loss: 0.4067 - val_recall: 0.2359
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3977 - recall: 0.2760 - val_loss: 0.4050 - val_recall: 0.2457
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3958 - recall: 0.2833 - val_loss: 0.4033 - val_recall: 0.2531
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3938 - recall: 0.2957 - val_loss: 0.4014 - val_recall: 0.2654
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3918 - recall: 0.3055 - val_loss: 0.3995 - val_recall: 0.2727
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3897 - recall: 0.3043 - val_loss: 0.3974 - val_recall: 0.2801
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3875 - recall: 0.3174 - val_loss: 0.3953 - val_recall: 0.2850
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3852 - recall: 0.3208 - val_loss: 0.3932 - val_recall: 0.2948
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3829 - recall: 0.3296 - val_loss: 0.3910 - val_recall: 0.3022
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3806 - recall: 0.3392 - val_loss: 0.3889 - val_recall: 0.3145
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3783 - recall: 0.3489 - val_loss: 0.3867 - val_recall: 0.3170
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3760 - recall: 0.3515 - val_loss: 0.3844 - val_recall: 0.3268
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3737 - recall: 0.3598 - val_loss: 0.3821 - val_recall: 0.3415
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3714 - recall: 0.3702 - val_loss: 0.3797 - val_recall: 0.3538
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3691 - recall: 0.3779 - val_loss: 0.3775 - val_recall: 0.3661
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3668 - recall: 0.3848 - val_loss: 0.3753 - val_recall: 0.3710
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3645 - recall: 0.3871 - val_loss: 0.3732 - val_recall: 0.3833
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3623 - recall: 0.3965 - val_loss: 0.3713 - val_recall: 0.3857
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3603 - recall: 0.4028 - val_loss: 0.3695 - val_recall: 0.3980

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(loss=sgd_history.history['loss'],val_loss=sgd_history.history['val_loss'])

• The loss in the validation and train sets is close and shows little noise

Recall

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_history.history['recall'],sgd_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = sgd_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 750us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using best as a threshold
y_val_pred = sgd_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 542us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train, y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val, y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.429272
Name: Neural Network (64,32,16,8) with SGD, dtype: float64
recall    0.398034
Name: Neural Network (64,32,16,8) with SGD, dtype: float64

Classification report

#classification report
print(classification_report(y_train, y_train_pred))

              precision    recall  f1-score   support

           0       0.87      0.97      0.91      4777
           1       0.76      0.43      0.55      1223

    accuracy                           0.86      6000
   macro avg       0.81      0.70      0.73      6000
weighted avg       0.85      0.86      0.84      6000

#classification report
print(classification_report(y_val, y_val_pred))

              precision    recall  f1-score   support

           0       0.86      0.97      0.91      1593
           1       0.75      0.40      0.52       407

    accuracy                           0.85      2000
   macro avg       0.80      0.68      0.72      2000
weighted avg       0.84      0.85      0.83      2000

Confusion matrix

make_confusion_matrix(y_train, y_train_pred)

make_confusion_matrix(y_val, y_val_pred)

Observations

• Recall for this model is very low, with training and validation both in the mid to upper 40's
• Let's add momentum and see if it helps.

Neural Network (64,32,16,8) with SGD + Momentum (0.9)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SGD + Momentum (0.9)"

#Initializing the neural network
sgd_momentum_model = Sequential()
sgd_momentum_model.add(Dense(64, activation='relu', input_dim = X_train.shape[1]))
sgd_momentum_model.add(Dense(32, activation='relu'))
sgd_momentum_model.add(Dense(16, activation='relu'))
sgd_momentum_model.add(Dense(8, activation='relu'))
sgd_momentum_model.add(Dense(1, activation = 'sigmoid'))

# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(momentum=0.9)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

check if this is the correct metric

## compile the model with binary cross entropy as loss function and recall as the metric.
sgd_momentum_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

sgd_momentum_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

check batch size and epochs

# Fitting the ANN
sgd_momentum_history = sgd_momentum_model.fit(
    X_train, y_train,
    batch_size=batch_size, 
    epochs=epochs,
    validation_data=(X_val,y_val),
    verbose=1
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5618 - recall: 0.0316 - val_loss: 0.4647 - val_recall: 0.0000e+00
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4473 - recall: 0.0000e+00 - val_loss: 0.4293 - val_recall: 0.0025
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4224 - recall: 0.0103 - val_loss: 0.4212 - val_recall: 0.1744
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4119 - recall: 0.1734 - val_loss: 0.4120 - val_recall: 0.2604
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4007 - recall: 0.2639 - val_loss: 0.3974 - val_recall: 0.3440
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3850 - recall: 0.3386 - val_loss: 0.3800 - val_recall: 0.4472
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3660 - recall: 0.4105 - val_loss: 0.3640 - val_recall: 0.4373
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3530 - recall: 0.4427 - val_loss: 0.3569 - val_recall: 0.4349
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3451 - recall: 0.4585 - val_loss: 0.3548 - val_recall: 0.4079
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3404 - recall: 0.4674 - val_loss: 0.3524 - val_recall: 0.4201
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3361 - recall: 0.4759 - val_loss: 0.3528 - val_recall: 0.4103
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3331 - recall: 0.4810 - val_loss: 0.3531 - val_recall: 0.4029
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3307 - recall: 0.4832 - val_loss: 0.3551 - val_recall: 0.3956
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3289 - recall: 0.4916 - val_loss: 0.3552 - val_recall: 0.3907
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3270 - recall: 0.4902 - val_loss: 0.3554 - val_recall: 0.4029
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3254 - recall: 0.4926 - val_loss: 0.3546 - val_recall: 0.4054
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3236 - recall: 0.4965 - val_loss: 0.3552 - val_recall: 0.3956
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3223 - recall: 0.4980 - val_loss: 0.3560 - val_recall: 0.3956
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3206 - recall: 0.4992 - val_loss: 0.3574 - val_recall: 0.3956
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3192 - recall: 0.5048 - val_loss: 0.3589 - val_recall: 0.3980
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3185 - recall: 0.5071 - val_loss: 0.3601 - val_recall: 0.3857
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3175 - recall: 0.5069 - val_loss: 0.3610 - val_recall: 0.3857
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3162 - recall: 0.5178 - val_loss: 0.3624 - val_recall: 0.3784
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3153 - recall: 0.5122 - val_loss: 0.3629 - val_recall: 0.3808
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3143 - recall: 0.5176 - val_loss: 0.3639 - val_recall: 0.3784
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3135 - recall: 0.5193 - val_loss: 0.3645 - val_recall: 0.3808
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3125 - recall: 0.5213 - val_loss: 0.3648 - val_recall: 0.3857
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3115 - recall: 0.5217 - val_loss: 0.3666 - val_recall: 0.3833
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3107 - recall: 0.5240 - val_loss: 0.3678 - val_recall: 0.3907
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3100 - recall: 0.5184 - val_loss: 0.3682 - val_recall: 0.3907
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3089 - recall: 0.5236 - val_loss: 0.3694 - val_recall: 0.3882
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3078 - recall: 0.5206 - val_loss: 0.3703 - val_recall: 0.3882
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3074 - recall: 0.5189 - val_loss: 0.3708 - val_recall: 0.3882
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3061 - recall: 0.5249 - val_loss: 0.3720 - val_recall: 0.3907
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3053 - recall: 0.5271 - val_loss: 0.3723 - val_recall: 0.3907
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3043 - recall: 0.5308 - val_loss: 0.3731 - val_recall: 0.3907
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3033 - recall: 0.5249 - val_loss: 0.3737 - val_recall: 0.3956
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3027 - recall: 0.5345 - val_loss: 0.3760 - val_recall: 0.3882
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3020 - recall: 0.5361 - val_loss: 0.3762 - val_recall: 0.3931
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3010 - recall: 0.5354 - val_loss: 0.3779 - val_recall: 0.3907
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3002 - recall: 0.5437 - val_loss: 0.3794 - val_recall: 0.3931
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2995 - recall: 0.5389 - val_loss: 0.3787 - val_recall: 0.3882
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2986 - recall: 0.5424 - val_loss: 0.3793 - val_recall: 0.3907
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2975 - recall: 0.5476 - val_loss: 0.3796 - val_recall: 0.3980
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2965 - recall: 0.5561 - val_loss: 0.3812 - val_recall: 0.4005
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2959 - recall: 0.5489 - val_loss: 0.3820 - val_recall: 0.4029
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2948 - recall: 0.5539 - val_loss: 0.3833 - val_recall: 0.4079
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2938 - recall: 0.5508 - val_loss: 0.3846 - val_recall: 0.4029
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2928 - recall: 0.5533 - val_loss: 0.3855 - val_recall: 0.4054
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2922 - recall: 0.5543 - val_loss: 0.3862 - val_recall: 0.4054

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(loss=sgd_momentum_history.history['loss'],val_loss=sgd_history.history['val_loss'])

• The loss in the validation and train sets is not close but shows little noise

Recall

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_momentum_history.history['recall'],sgd_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = sgd_momentum_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 716us/step

array([[False],
       [False],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using best as a threshold
y_val_pred = sgd_momentum_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 543us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train, y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val, y_val_pred)


print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.518397
Name: Neural Network (64,32,16,8) with SGD + Momentum (0.9), dtype: float64
recall    0.405405
Name: Neural Network (64,32,16,8) with SGD + Momentum (0.9), dtype: float64

Classification report

#classification report
print(classification_report(y_train, y_train_pred))

              precision    recall  f1-score   support

           0       0.89      0.98      0.93      4777
           1       0.86      0.52      0.65      1223

    accuracy                           0.89      6000
   macro avg       0.88      0.75      0.79      6000
weighted avg       0.88      0.89      0.87      6000

#classification report
print(classification_report(y_val, y_val_pred))

              precision    recall  f1-score   support

           0       0.86      0.97      0.91      1593
           1       0.77      0.41      0.53       407

    accuracy                           0.85      2000
   macro avg       0.82      0.69      0.72      2000
weighted avg       0.85      0.85      0.84      2000

Confusion matrix

make_confusion_matrix(y_train, y_train_pred)

make_confusion_matrix(y_val, y_val_pred)

Observations

• Momentum improved recall performance on the traing set, but no improvemnt in validation.
• Let's try Adam this time.

Model Performance Improvement

Neural Network (64,32,16,8) with Adam Optimizer

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with Adam"

#Initializing the neural network
adam_model = Sequential()
adam_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_model.add(Dense(32,activation='relu'))
adam_model.add(Dense(16,activation='relu'))
adam_model.add(Dense(8,activation='relu'))
adam_model.add(Dense(1, activation = 'sigmoid'))

#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN
adam_history = adam_model.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    validation_data=(X_val,y_val),
    verbose=1
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5593 - recall: 0.0227 - val_loss: 0.4319 - val_recall: 0.0074
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4249 - recall: 0.0466 - val_loss: 0.4178 - val_recall: 0.1695
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4067 - recall: 0.1933 - val_loss: 0.3960 - val_recall: 0.3243
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3789 - recall: 0.3813 - val_loss: 0.3712 - val_recall: 0.4152
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3547 - recall: 0.4507 - val_loss: 0.3596 - val_recall: 0.4423
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3428 - recall: 0.4688 - val_loss: 0.3554 - val_recall: 0.4570
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3355 - recall: 0.4841 - val_loss: 0.3535 - val_recall: 0.4595
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3304 - recall: 0.4894 - val_loss: 0.3531 - val_recall: 0.4472
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3260 - recall: 0.5006 - val_loss: 0.3532 - val_recall: 0.4349
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3222 - recall: 0.5057 - val_loss: 0.3528 - val_recall: 0.4398
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3185 - recall: 0.5157 - val_loss: 0.3526 - val_recall: 0.4423
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3156 - recall: 0.5211 - val_loss: 0.3530 - val_recall: 0.4423
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3126 - recall: 0.5262 - val_loss: 0.3538 - val_recall: 0.4496
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3104 - recall: 0.5305 - val_loss: 0.3533 - val_recall: 0.4545
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3085 - recall: 0.5389 - val_loss: 0.3538 - val_recall: 0.4521
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3068 - recall: 0.5378 - val_loss: 0.3544 - val_recall: 0.4570
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3048 - recall: 0.5410 - val_loss: 0.3553 - val_recall: 0.4545
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3027 - recall: 0.5511 - val_loss: 0.3550 - val_recall: 0.4545
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3003 - recall: 0.5499 - val_loss: 0.3557 - val_recall: 0.4570
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2982 - recall: 0.5538 - val_loss: 0.3569 - val_recall: 0.4521
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2963 - recall: 0.5538 - val_loss: 0.3578 - val_recall: 0.4447
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2941 - recall: 0.5571 - val_loss: 0.3579 - val_recall: 0.4447
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2924 - recall: 0.5578 - val_loss: 0.3580 - val_recall: 0.4472
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2901 - recall: 0.5640 - val_loss: 0.3587 - val_recall: 0.4545
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2879 - recall: 0.5724 - val_loss: 0.3588 - val_recall: 0.4545
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2857 - recall: 0.5755 - val_loss: 0.3599 - val_recall: 0.4570
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2843 - recall: 0.5718 - val_loss: 0.3611 - val_recall: 0.4595
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2818 - recall: 0.5763 - val_loss: 0.3620 - val_recall: 0.4644
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2797 - recall: 0.5771 - val_loss: 0.3635 - val_recall: 0.4619
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2772 - recall: 0.5857 - val_loss: 0.3647 - val_recall: 0.4668
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2747 - recall: 0.5888 - val_loss: 0.3663 - val_recall: 0.4717
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2731 - recall: 0.5948 - val_loss: 0.3689 - val_recall: 0.4644
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2714 - recall: 0.5935 - val_loss: 0.3717 - val_recall: 0.4717
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2692 - recall: 0.5912 - val_loss: 0.3733 - val_recall: 0.4717
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2673 - recall: 0.5944 - val_loss: 0.3760 - val_recall: 0.4668
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2658 - recall: 0.6120 - val_loss: 0.3779 - val_recall: 0.4717
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2630 - recall: 0.6123 - val_loss: 0.3796 - val_recall: 0.4742
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2616 - recall: 0.6066 - val_loss: 0.3809 - val_recall: 0.4889
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2595 - recall: 0.6215 - val_loss: 0.3831 - val_recall: 0.4742
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2572 - recall: 0.6280 - val_loss: 0.3846 - val_recall: 0.4816
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2556 - recall: 0.6305 - val_loss: 0.3867 - val_recall: 0.4791
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2530 - recall: 0.6329 - val_loss: 0.3907 - val_recall: 0.4865
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2513 - recall: 0.6369 - val_loss: 0.3963 - val_recall: 0.4914
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2506 - recall: 0.6357 - val_loss: 0.3959 - val_recall: 0.4865
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2478 - recall: 0.6400 - val_loss: 0.3985 - val_recall: 0.4791
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2454 - recall: 0.6426 - val_loss: 0.4030 - val_recall: 0.4865
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2441 - recall: 0.6492 - val_loss: 0.4049 - val_recall: 0.4742
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2428 - recall: 0.6544 - val_loss: 0.4076 - val_recall: 0.4693
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2411 - recall: 0.6521 - val_loss: 0.4110 - val_recall: 0.4742
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2385 - recall: 0.6549 - val_loss: 0.4141 - val_recall: 0.4791

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history.history['loss'],adam_history.history['val_loss'])

• The loss in the validation and train sets is not close past epoch 5 and shows little noise

Recall

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history.history['recall'],adam_history.history['val_recall'])

#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 744us/step

array([[False],
       [False],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 567us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.651676
Name: Neural Network (64,32,16,8) with Adam, dtype: float64
recall    0.479115
Name: Neural Network (64,32,16,8) with Adam, dtype: float64

Classification report

#lassification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.92      0.97      0.94      4777
           1       0.85      0.65      0.74      1223

    accuracy                           0.91      6000
   macro avg       0.88      0.81      0.84      6000
weighted avg       0.90      0.91      0.90      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.88      0.94      0.91      1593
           1       0.67      0.48      0.56       407

    accuracy                           0.85      2000
   macro avg       0.77      0.71      0.73      2000
weighted avg       0.83      0.85      0.84      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Training recall improved, but validation got worse.
• Let's try increasing complexity with more layers to see what happens.

Neural Network (256,128,64,32,16,8) with Adam Optimizer

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (256,128,64,32,16,8) with Adam"

#Initializing the neural network
adam_model2 = Sequential()
adam_model2.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
adam_model2.add(Dense(128,activation='relu'))
adam_model2.add(Dense(64,activation='relu'))
adam_model2.add(Dense(32,activation='relu'))
adam_model2.add(Dense(16,activation='relu'))
adam_model2.add(Dense(8,activation='relu'))
adam_model2.add(Dense(1, activation = 'sigmoid'))

#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_model2.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 256)            │         3,072 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 128)            │        32,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 64)             │         8,256 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_5 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_6 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 46,977 (183.50 KB)

 Trainable params: 46,977 (183.50 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN
adam_history2 = adam_model2.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    validation_data=(X_val,y_val),
    verbose=1
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.5675 - recall: 0.1212 - val_loss: 0.4238 - val_recall: 0.2260
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4069 - recall: 0.2640 - val_loss: 0.3940 - val_recall: 0.3857
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3706 - recall: 0.3913 - val_loss: 0.3704 - val_recall: 0.3661
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3492 - recall: 0.4323 - val_loss: 0.3654 - val_recall: 0.3292
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3398 - recall: 0.4311 - val_loss: 0.3629 - val_recall: 0.3292
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3334 - recall: 0.4415 - val_loss: 0.3604 - val_recall: 0.3243
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3285 - recall: 0.4303 - val_loss: 0.3608 - val_recall: 0.3096
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3245 - recall: 0.4324 - val_loss: 0.3626 - val_recall: 0.2899
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3203 - recall: 0.4293 - val_loss: 0.3617 - val_recall: 0.3243
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3147 - recall: 0.4361 - val_loss: 0.3626 - val_recall: 0.3489
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3100 - recall: 0.4435 - val_loss: 0.3635 - val_recall: 0.3317
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3044 - recall: 0.4502 - val_loss: 0.3677 - val_recall: 0.3440
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2969 - recall: 0.4680 - val_loss: 0.3692 - val_recall: 0.3587
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2902 - recall: 0.4755 - val_loss: 0.3720 - val_recall: 0.3563
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2831 - recall: 0.4944 - val_loss: 0.3791 - val_recall: 0.3563
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2766 - recall: 0.4996 - val_loss: 0.3816 - val_recall: 0.3342
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2713 - recall: 0.4892 - val_loss: 0.3921 - val_recall: 0.3489
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2676 - recall: 0.5118 - val_loss: 0.3946 - val_recall: 0.3415
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2607 - recall: 0.5508 - val_loss: 0.3996 - val_recall: 0.2998
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2548 - recall: 0.5692 - val_loss: 0.4019 - val_recall: 0.3980
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2522 - recall: 0.5747 - val_loss: 0.4142 - val_recall: 0.4570
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2542 - recall: 0.6061 - val_loss: 0.4295 - val_recall: 0.4545
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2489 - recall: 0.6085 - val_loss: 0.4745 - val_recall: 0.4791
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2576 - recall: 0.5906 - val_loss: 0.4514 - val_recall: 0.4668
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2497 - recall: 0.6210 - val_loss: 0.4547 - val_recall: 0.4693
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2400 - recall: 0.6411 - val_loss: 0.4676 - val_recall: 0.4521
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2371 - recall: 0.6449 - val_loss: 0.4509 - val_recall: 0.4275
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2333 - recall: 0.6390 - val_loss: 0.4819 - val_recall: 0.4398
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2187 - recall: 0.6514 - val_loss: 0.4898 - val_recall: 0.4079
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2111 - recall: 0.6668 - val_loss: 0.4833 - val_recall: 0.4079
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2036 - recall: 0.6893 - val_loss: 0.5300 - val_recall: 0.3415
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1947 - recall: 0.6942 - val_loss: 0.5567 - val_recall: 0.4226
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1857 - recall: 0.7270 - val_loss: 0.5749 - val_recall: 0.3636
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1907 - recall: 0.7045 - val_loss: 0.5994 - val_recall: 0.3686
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1944 - recall: 0.6973 - val_loss: 0.5827 - val_recall: 0.4398
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1739 - recall: 0.7377 - val_loss: 0.5999 - val_recall: 0.4398
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1705 - recall: 0.7551 - val_loss: 0.6237 - val_recall: 0.4226
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1610 - recall: 0.7724 - val_loss: 0.6634 - val_recall: 0.4251
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1581 - recall: 0.7681 - val_loss: 0.6665 - val_recall: 0.4742
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1606 - recall: 0.7885 - val_loss: 0.6949 - val_recall: 0.5037
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1610 - recall: 0.7880 - val_loss: 0.6720 - val_recall: 0.5012
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1475 - recall: 0.7868 - val_loss: 0.7460 - val_recall: 0.5725
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1567 - recall: 0.7951 - val_loss: 0.7438 - val_recall: 0.5405
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1553 - recall: 0.7984 - val_loss: 0.7154 - val_recall: 0.5233
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1333 - recall: 0.8104 - val_loss: 0.7832 - val_recall: 0.5627
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1331 - recall: 0.8246 - val_loss: 0.8302 - val_recall: 0.5749
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1460 - recall: 0.8182 - val_loss: 0.7945 - val_recall: 0.5749
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1489 - recall: 0.8245 - val_loss: 0.8297 - val_recall: 0.5184
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1236 - recall: 0.8519 - val_loss: 0.9069 - val_recall: 0.4742
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1226 - recall: 0.8566 - val_loss: 0.9586 - val_recall: 0.4914

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history2.history['loss'],adam_history2.history['val_loss'])

• The loss in the validation and train sets diverges drastically and begins showing noise after epoch 20

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history2.history['recall'],adam_history2.history['val_recall'])

#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model2.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 913us/step

array([[False],
       [ True],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model2.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.806214
Name: Neural Network (256,128,64,32,16,8) with Adam, dtype: float64
recall    0.4914
Name: Neural Network (256,128,64,32,16,8) with Adam, dtype: float64

Classification report

#lassification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.95      0.98      0.97      4777
           1       0.91      0.81      0.86      1223

    accuracy                           0.94      6000
   macro avg       0.93      0.89      0.91      6000
weighted avg       0.94      0.94      0.94      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.88      0.92      0.90      1593
           1       0.61      0.49      0.54       407

    accuracy                           0.83      2000
   macro avg       0.74      0.71      0.72      2000
weighted avg       0.82      0.83      0.83      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Training Recall improved quite a bit, but validation not so much.
• Training is probably overfitting the data.
• Let's see if adding a learning rate improves anything. We'll go with .001 to start

Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)"

#Initializing the neural network
adam_model3 = Sequential()
adam_model3.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
adam_model3.add(Dense(128,activation='relu'))
adam_model3.add(Dense(64,activation='relu'))
adam_model3.add(Dense(128,activation='relu'))
adam_model3.add(Dense(64,activation='relu'))
adam_model3.add(Dense(8,activation='relu'))
adam_model3.add(Dense(1, activation = 'sigmoid'))

#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(0.001)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 10,565 (41.27 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

 Optimizer params: 7,044 (27.52 KB)

Loss function

#Fitting the ANN
adam_history3 = adam_model3.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    validation_data=(X_val,y_val),
    verbose=1
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - loss: 0.5314 - recall: 0.0664 - val_loss: 0.4201 - val_recall: 0.1892
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3925 - recall: 0.3479 - val_loss: 0.3848 - val_recall: 0.3464
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3557 - recall: 0.4013 - val_loss: 0.3611 - val_recall: 0.3882
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3372 - recall: 0.4386 - val_loss: 0.3598 - val_recall: 0.3538
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3301 - recall: 0.4380 - val_loss: 0.3578 - val_recall: 0.3735
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3252 - recall: 0.4473 - val_loss: 0.3581 - val_recall: 0.3538
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3180 - recall: 0.4527 - val_loss: 0.3597 - val_recall: 0.3636
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3155 - recall: 0.4541 - val_loss: 0.3641 - val_recall: 0.3784
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3091 - recall: 0.4625 - val_loss: 0.3668 - val_recall: 0.3342
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3012 - recall: 0.4650 - val_loss: 0.3737 - val_recall: 0.3735
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2966 - recall: 0.4745 - val_loss: 0.3770 - val_recall: 0.3735
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2897 - recall: 0.4795 - val_loss: 0.3818 - val_recall: 0.3956
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2851 - recall: 0.4917 - val_loss: 0.3992 - val_recall: 0.4005
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2803 - recall: 0.5023 - val_loss: 0.3945 - val_recall: 0.3636
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2724 - recall: 0.5077 - val_loss: 0.4352 - val_recall: 0.4791
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2734 - recall: 0.5178 - val_loss: 0.4359 - val_recall: 0.4717
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2688 - recall: 0.5218 - val_loss: 0.4138 - val_recall: 0.4545
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2602 - recall: 0.5416 - val_loss: 0.4592 - val_recall: 0.4693
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2544 - recall: 0.5503 - val_loss: 0.4677 - val_recall: 0.4447
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2452 - recall: 0.5744 - val_loss: 0.5054 - val_recall: 0.4742
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2462 - recall: 0.5372 - val_loss: 0.5107 - val_recall: 0.5135
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2439 - recall: 0.5834 - val_loss: 0.5499 - val_recall: 0.5160
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2273 - recall: 0.6164 - val_loss: 0.5675 - val_recall: 0.5184
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2185 - recall: 0.6245 - val_loss: 0.6031 - val_recall: 0.5307
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2087 - recall: 0.6313 - val_loss: 0.6230 - val_recall: 0.5283
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2053 - recall: 0.6288 - val_loss: 0.6567 - val_recall: 0.5160
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1957 - recall: 0.6672 - val_loss: 0.6784 - val_recall: 0.4963
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1926 - recall: 0.6818 - val_loss: 0.6659 - val_recall: 0.4816
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1961 - recall: 0.6715 - val_loss: 0.6566 - val_recall: 0.5111
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1955 - recall: 0.7014 - val_loss: 0.6680 - val_recall: 0.4889
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1947 - recall: 0.7060 - val_loss: 0.7147 - val_recall: 0.4816
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1666 - recall: 0.7317 - val_loss: 0.7086 - val_recall: 0.4988
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1648 - recall: 0.7386 - val_loss: 0.7528 - val_recall: 0.4644
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1661 - recall: 0.7511 - val_loss: 0.7301 - val_recall: 0.4496
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1459 - recall: 0.7724 - val_loss: 0.7509 - val_recall: 0.3882
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1518 - recall: 0.7758 - val_loss: 0.6955 - val_recall: 0.4300
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1515 - recall: 0.7655 - val_loss: 0.7487 - val_recall: 0.4324
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1509 - recall: 0.7836 - val_loss: 0.7408 - val_recall: 0.4128
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1314 - recall: 0.7919 - val_loss: 0.7819 - val_recall: 0.4275
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1308 - recall: 0.8074 - val_loss: 0.9014 - val_recall: 0.4251
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1220 - recall: 0.8158 - val_loss: 0.8531 - val_recall: 0.3661
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1145 - recall: 0.8257 - val_loss: 0.8731 - val_recall: 0.3661
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0938 - recall: 0.8484 - val_loss: 1.0320 - val_recall: 0.3710
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0990 - recall: 0.8731 - val_loss: 1.1081 - val_recall: 0.3833
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1201 - recall: 0.8200 - val_loss: 1.0196 - val_recall: 0.4251
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1155 - recall: 0.8386 - val_loss: 0.9493 - val_recall: 0.4423
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1038 - recall: 0.8656 - val_loss: 0.9677 - val_recall: 0.4742
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0958 - recall: 0.8803 - val_loss: 0.9661 - val_recall: 0.4693
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0862 - recall: 0.8984 - val_loss: 1.0386 - val_recall: 0.4349
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0950 - recall: 0.8892 - val_loss: 0.9834 - val_recall: 0.4570

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history3.history['loss'],adam_history3.history['val_loss'])

• The loss in the validation and train sets diverges drastically and begins showing noise after epoch 12

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history3.history['recall'],adam_history3.history['val_recall'])

#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model3.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 934us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model3.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 629us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.876533
Name: Neural Network (256,128,64,128,64,8) with Adam + LR (0.001), dtype: float64
recall    0.457002
Name: Neural Network (256,128,64,128,64,8) with Adam + LR (0.001), dtype: float64

Classification report

#lassification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.97      0.97      0.97      4777
           1       0.87      0.88      0.87      1223

    accuracy                           0.95      6000
   macro avg       0.92      0.92      0.92      6000
weighted avg       0.95      0.95      0.95      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.87      0.89      0.88      1593
           1       0.52      0.46      0.49       407

    accuracy                           0.80      2000
   macro avg       0.69      0.67      0.68      2000
weighted avg       0.79      0.80      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• About the same resuts as previous model, with high recall on training data but low on validation.
• Lets resimplify the model and add Dropout.

Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)"

#Initializing the neural network
adam_dropout_model = Sequential()
adam_dropout_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model.add(Dropout(0.4))
adam_dropout_model.add(Dense(32,activation='relu'))
adam_dropout_model.add(Dropout(0.3))
adam_dropout_model.add(Dense(16,activation='relu'))
adam_dropout_model.add(Dropout(0.2))
adam_dropout_model.add(Dense(8,activation='relu'))
adam_dropout_model.add(Dropout(0.1))
adam_dropout_model.add(Dense(1, activation = 'sigmoid'))

#Cuse Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_dropout_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN 
adam_dropout_history = adam_dropout_model.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.6397 - recall: 0.1852 - val_loss: 0.4683 - val_recall: 0.0000e+00
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4878 - recall: 0.0000e+00 - val_loss: 0.4347 - val_recall: 0.0000e+00
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4600 - recall: 1.3815e-04 - val_loss: 0.4331 - val_recall: 0.0000e+00
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4593 - recall: 0.0059 - val_loss: 0.4305 - val_recall: 0.0000e+00
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4513 - recall: 0.0253 - val_loss: 0.4259 - val_recall: 0.1204
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4426 - recall: 0.1141 - val_loss: 0.4191 - val_recall: 0.2703
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4381 - recall: 0.1852 - val_loss: 0.4107 - val_recall: 0.3022
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4196 - recall: 0.2449 - val_loss: 0.4028 - val_recall: 0.3194
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4148 - recall: 0.2763 - val_loss: 0.3937 - val_recall: 0.3612
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4140 - recall: 0.2679 - val_loss: 0.3880 - val_recall: 0.4152
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4043 - recall: 0.3106 - val_loss: 0.3813 - val_recall: 0.3563
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3970 - recall: 0.2986 - val_loss: 0.3747 - val_recall: 0.3857
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3954 - recall: 0.3161 - val_loss: 0.3727 - val_recall: 0.3661
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3992 - recall: 0.3361 - val_loss: 0.3690 - val_recall: 0.3710
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3845 - recall: 0.3516 - val_loss: 0.3641 - val_recall: 0.3882
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3795 - recall: 0.3776 - val_loss: 0.3624 - val_recall: 0.4103
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3846 - recall: 0.3780 - val_loss: 0.3614 - val_recall: 0.3980
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3769 - recall: 0.3654 - val_loss: 0.3578 - val_recall: 0.4054
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3695 - recall: 0.3802 - val_loss: 0.3575 - val_recall: 0.3833
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3715 - recall: 0.3755 - val_loss: 0.3577 - val_recall: 0.3956
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3688 - recall: 0.3859 - val_loss: 0.3543 - val_recall: 0.4251
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3633 - recall: 0.4082 - val_loss: 0.3550 - val_recall: 0.4251
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3644 - recall: 0.4186 - val_loss: 0.3537 - val_recall: 0.4668
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3709 - recall: 0.4148 - val_loss: 0.3529 - val_recall: 0.4324
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3649 - recall: 0.4213 - val_loss: 0.3520 - val_recall: 0.4668
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3671 - recall: 0.4268 - val_loss: 0.3524 - val_recall: 0.4472
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3645 - recall: 0.4198 - val_loss: 0.3511 - val_recall: 0.4496
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3627 - recall: 0.4296 - val_loss: 0.3528 - val_recall: 0.4767
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3538 - recall: 0.4636 - val_loss: 0.3524 - val_recall: 0.4693
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3610 - recall: 0.4488 - val_loss: 0.3510 - val_recall: 0.4619
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3580 - recall: 0.4316 - val_loss: 0.3511 - val_recall: 0.4619
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3573 - recall: 0.4417 - val_loss: 0.3534 - val_recall: 0.4496
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3617 - recall: 0.4456 - val_loss: 0.3525 - val_recall: 0.4570
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3525 - recall: 0.4609 - val_loss: 0.3519 - val_recall: 0.4152
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3525 - recall: 0.4288 - val_loss: 0.3517 - val_recall: 0.4644
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3606 - recall: 0.4368 - val_loss: 0.3526 - val_recall: 0.4300
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3591 - recall: 0.4310 - val_loss: 0.3517 - val_recall: 0.4545
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3555 - recall: 0.4570 - val_loss: 0.3509 - val_recall: 0.4423
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3582 - recall: 0.4433 - val_loss: 0.3509 - val_recall: 0.4595
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3497 - recall: 0.4538 - val_loss: 0.3505 - val_recall: 0.4816
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3569 - recall: 0.4594 - val_loss: 0.3491 - val_recall: 0.4889
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3532 - recall: 0.4505 - val_loss: 0.3491 - val_recall: 0.4840
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3480 - recall: 0.4535 - val_loss: 0.3519 - val_recall: 0.4619
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3506 - recall: 0.4612 - val_loss: 0.3510 - val_recall: 0.4742
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3542 - recall: 0.4575 - val_loss: 0.3516 - val_recall: 0.4717
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3430 - recall: 0.4616 - val_loss: 0.3512 - val_recall: 0.4619
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3541 - recall: 0.4611 - val_loss: 0.3513 - val_recall: 0.4619
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3543 - recall: 0.4541 - val_loss: 0.3511 - val_recall: 0.4742
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3468 - recall: 0.4609 - val_loss: 0.3502 - val_recall: 0.4693
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3475 - recall: 0.4463 - val_loss: 0.3512 - val_recall: 0.4742

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history.history['loss'],adam_dropout_history.history['val_loss'])

• The loss in the validation and train sets converges again around epoch 26, but traing set shows a bit of noise due to dropout

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history.history['recall'],adam_dropout_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 727us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 568us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.510221
Name: Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
recall    0.474201
Name: Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.88      0.96      0.92      4777
           1       0.78      0.51      0.62      1223

    accuracy                           0.87      6000
   macro avg       0.83      0.74      0.77      6000
weighted avg       0.86      0.87      0.86      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.88      0.96      0.91      1593
           1       0.73      0.47      0.58       407

    accuracy                           0.86      2000
   macro avg       0.80      0.71      0.74      2000
weighted avg       0.85      0.86      0.85      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Terrible recall performance, so lets simplify and see.
• Accuracy and Precision seem to have improved, but we're not optimizing for those.

Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)"

#Initializing the neural network
adam_dropout_model2 = Sequential()
adam_dropout_model2.add(Dense(32,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model2.add(Dropout(0.4))
adam_dropout_model2.add(Dense(16,activation='relu'))
adam_dropout_model2.add(Dropout(0.3))
adam_dropout_model2.add(Dense(8,activation='relu'))
adam_dropout_model2.add(Dropout(0.2))
adam_dropout_model2.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_dropout_model2.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 32)             │           384 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 1,057 (4.13 KB)

 Trainable params: 1,057 (4.13 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN with batch_size = 32 and 100 epochs
adam_dropout_history2 = adam_dropout_model2.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.7092 - recall: 0.5137 - val_loss: 0.5181 - val_recall: 0.0000e+00
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5431 - recall: 0.0227 - val_loss: 0.4635 - val_recall: 0.0000e+00
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4995 - recall: 0.0240 - val_loss: 0.4447 - val_recall: 0.0074
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4850 - recall: 0.0316 - val_loss: 0.4354 - val_recall: 0.0197
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4705 - recall: 0.0701 - val_loss: 0.4313 - val_recall: 0.0221
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4627 - recall: 0.0546 - val_loss: 0.4282 - val_recall: 0.0344
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4590 - recall: 0.0693 - val_loss: 0.4256 - val_recall: 0.0516
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4503 - recall: 0.0872 - val_loss: 0.4226 - val_recall: 0.0565
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4655 - recall: 0.0922 - val_loss: 0.4205 - val_recall: 0.0639
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4509 - recall: 0.1309 - val_loss: 0.4167 - val_recall: 0.0786
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4451 - recall: 0.1626 - val_loss: 0.4113 - val_recall: 0.1400
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4435 - recall: 0.1768 - val_loss: 0.4068 - val_recall: 0.1843
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4318 - recall: 0.2181 - val_loss: 0.4014 - val_recall: 0.2604
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4233 - recall: 0.2409 - val_loss: 0.3985 - val_recall: 0.2457
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4240 - recall: 0.2342 - val_loss: 0.3937 - val_recall: 0.2703
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4214 - recall: 0.2818 - val_loss: 0.3893 - val_recall: 0.2899
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4188 - recall: 0.2649 - val_loss: 0.3861 - val_recall: 0.2678
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4160 - recall: 0.2827 - val_loss: 0.3840 - val_recall: 0.2924
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4144 - recall: 0.2873 - val_loss: 0.3806 - val_recall: 0.2948
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4141 - recall: 0.2934 - val_loss: 0.3784 - val_recall: 0.3219
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4094 - recall: 0.2991 - val_loss: 0.3764 - val_recall: 0.3415
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4068 - recall: 0.3224 - val_loss: 0.3732 - val_recall: 0.3857
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4111 - recall: 0.3443 - val_loss: 0.3707 - val_recall: 0.3563
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4054 - recall: 0.3294 - val_loss: 0.3686 - val_recall: 0.3710
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3997 - recall: 0.3212 - val_loss: 0.3686 - val_recall: 0.3538
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4081 - recall: 0.3187 - val_loss: 0.3671 - val_recall: 0.3808
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3942 - recall: 0.3328 - val_loss: 0.3672 - val_recall: 0.3661
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3865 - recall: 0.3326 - val_loss: 0.3654 - val_recall: 0.3661
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3918 - recall: 0.3440 - val_loss: 0.3641 - val_recall: 0.3636
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3867 - recall: 0.3767 - val_loss: 0.3626 - val_recall: 0.3612
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3874 - recall: 0.3537 - val_loss: 0.3619 - val_recall: 0.3636
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3895 - recall: 0.3412 - val_loss: 0.3618 - val_recall: 0.3538
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3877 - recall: 0.3362 - val_loss: 0.3596 - val_recall: 0.3636
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3877 - recall: 0.3586 - val_loss: 0.3593 - val_recall: 0.3538
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3841 - recall: 0.3439 - val_loss: 0.3596 - val_recall: 0.3489
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3807 - recall: 0.3457 - val_loss: 0.3578 - val_recall: 0.3587
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3790 - recall: 0.3639 - val_loss: 0.3572 - val_recall: 0.3710
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3849 - recall: 0.3690 - val_loss: 0.3583 - val_recall: 0.3514
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3770 - recall: 0.3875 - val_loss: 0.3556 - val_recall: 0.3686
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3788 - recall: 0.3844 - val_loss: 0.3559 - val_recall: 0.3710
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3819 - recall: 0.3742 - val_loss: 0.3573 - val_recall: 0.3636
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3745 - recall: 0.3704 - val_loss: 0.3549 - val_recall: 0.3710
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3722 - recall: 0.3902 - val_loss: 0.3545 - val_recall: 0.3735
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3778 - recall: 0.3798 - val_loss: 0.3553 - val_recall: 0.3661
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3728 - recall: 0.3964 - val_loss: 0.3553 - val_recall: 0.3563
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3818 - recall: 0.3736 - val_loss: 0.3547 - val_recall: 0.3735
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3732 - recall: 0.3956 - val_loss: 0.3535 - val_recall: 0.3980
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3705 - recall: 0.4011 - val_loss: 0.3551 - val_recall: 0.3735
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3699 - recall: 0.3685 - val_loss: 0.3544 - val_recall: 0.3833
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3670 - recall: 0.3782 - val_loss: 0.3537 - val_recall: 0.3833

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history2.history['loss'],adam_dropout_history2.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and some noise in both sets though much more in triang due to dropout

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history2.history['recall'],adam_dropout_history2.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model2.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model2.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 544us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.420278
Name: Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2), dtype: float64
recall    0.383292
Name: Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2), dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.87      0.98      0.92      4777
           1       0.81      0.42      0.55      1223

    accuracy                           0.86      6000
   macro avg       0.84      0.70      0.74      6000
weighted avg       0.86      0.86      0.84      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.86      0.97      0.91      1593
           1       0.77      0.38      0.51       407

    accuracy                           0.85      2000
   macro avg       0.82      0.68      0.71      2000
weighted avg       0.84      0.85      0.83      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Still pretty terrible performance on recall.
• Lets simplify again and raise dropout

Neural Network (64,32) with Adam & Dropout (0.5, 0.5)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32) with Adam & Dropout (0.5, 0.5)"

#Initializing the neural network
adam_dropout_model3 = Sequential()
adam_dropout_model3.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model3.add(Dropout(0.5))
adam_dropout_model3.add(Dense(32,activation='relu'))
adam_dropout_model3.add(Dropout(0.5))
adam_dropout_model3.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_dropout_model3.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │            33 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 2,881 (11.25 KB)

 Trainable params: 2,881 (11.25 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN with batch_size = 32 and 100 epochs
adam_dropout_history3 = adam_dropout_model3.fit(
    X_train,y_train,
    batch_size=32, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4951 - recall: 0.0881 - val_loss: 0.4170 - val_recall: 0.0737
Epoch 2/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4314 - recall: 0.1775 - val_loss: 0.3998 - val_recall: 0.1695
Epoch 3/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4235 - recall: 0.1973 - val_loss: 0.3823 - val_recall: 0.2162
Epoch 4/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4125 - recall: 0.2549 - val_loss: 0.3717 - val_recall: 0.3022
Epoch 5/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3973 - recall: 0.2854 - val_loss: 0.3701 - val_recall: 0.2678
Epoch 6/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3948 - recall: 0.3105 - val_loss: 0.3707 - val_recall: 0.2924
Epoch 7/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3886 - recall: 0.2783 - val_loss: 0.3664 - val_recall: 0.3391
Epoch 8/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3782 - recall: 0.3142 - val_loss: 0.3681 - val_recall: 0.3268
Epoch 9/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3789 - recall: 0.3132 - val_loss: 0.3612 - val_recall: 0.3145
Epoch 10/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3797 - recall: 0.3382 - val_loss: 0.3599 - val_recall: 0.3243
Epoch 11/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3762 - recall: 0.3512 - val_loss: 0.3661 - val_recall: 0.2752
Epoch 12/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3904 - recall: 0.3121 - val_loss: 0.3698 - val_recall: 0.3194
Epoch 13/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3696 - recall: 0.3453 - val_loss: 0.3655 - val_recall: 0.3268
Epoch 14/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3751 - recall: 0.3487 - val_loss: 0.3666 - val_recall: 0.2899
Epoch 15/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3663 - recall: 0.3359 - val_loss: 0.3636 - val_recall: 0.3096
Epoch 16/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3811 - recall: 0.3266 - val_loss: 0.3664 - val_recall: 0.2850
Epoch 17/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3779 - recall: 0.3278 - val_loss: 0.3656 - val_recall: 0.3292
Epoch 18/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3779 - recall: 0.3248 - val_loss: 0.3630 - val_recall: 0.3538
Epoch 19/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3777 - recall: 0.3231 - val_loss: 0.3649 - val_recall: 0.2875
Epoch 20/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3753 - recall: 0.3257 - val_loss: 0.3641 - val_recall: 0.3096
Epoch 21/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3719 - recall: 0.3760 - val_loss: 0.3647 - val_recall: 0.2973
Epoch 22/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3685 - recall: 0.3576 - val_loss: 0.3618 - val_recall: 0.3440
Epoch 23/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3772 - recall: 0.3307 - val_loss: 0.3645 - val_recall: 0.3391
Epoch 24/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3743 - recall: 0.3635 - val_loss: 0.3709 - val_recall: 0.2776
Epoch 25/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3676 - recall: 0.3310 - val_loss: 0.3627 - val_recall: 0.3170
Epoch 26/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3768 - recall: 0.3379 - val_loss: 0.3614 - val_recall: 0.3415
Epoch 27/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3776 - recall: 0.3282 - val_loss: 0.3610 - val_recall: 0.3292
Epoch 28/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3676 - recall: 0.3543 - val_loss: 0.3610 - val_recall: 0.3759
Epoch 29/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3717 - recall: 0.3546 - val_loss: 0.3621 - val_recall: 0.3538
Epoch 30/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3728 - recall: 0.3364 - val_loss: 0.3629 - val_recall: 0.3514
Epoch 31/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3662 - recall: 0.3522 - val_loss: 0.3628 - val_recall: 0.3612
Epoch 32/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3711 - recall: 0.3490 - val_loss: 0.3612 - val_recall: 0.3440
Epoch 33/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3718 - recall: 0.3476 - val_loss: 0.3629 - val_recall: 0.3612
Epoch 34/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3807 - recall: 0.3550 - val_loss: 0.3613 - val_recall: 0.3022
Epoch 35/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3637 - recall: 0.3595 - val_loss: 0.3635 - val_recall: 0.2998
Epoch 36/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3708 - recall: 0.3556 - val_loss: 0.3647 - val_recall: 0.3710
Epoch 37/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3635 - recall: 0.3807 - val_loss: 0.3591 - val_recall: 0.3808
Epoch 38/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3731 - recall: 0.3692 - val_loss: 0.3609 - val_recall: 0.3538
Epoch 39/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3653 - recall: 0.3576 - val_loss: 0.3652 - val_recall: 0.3464
Epoch 40/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3758 - recall: 0.3405 - val_loss: 0.3612 - val_recall: 0.3464
Epoch 41/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3652 - recall: 0.3507 - val_loss: 0.3660 - val_recall: 0.3489
Epoch 42/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3725 - recall: 0.3480 - val_loss: 0.3632 - val_recall: 0.3857
Epoch 43/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3792 - recall: 0.3132 - val_loss: 0.3689 - val_recall: 0.3170
Epoch 44/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3752 - recall: 0.3390 - val_loss: 0.3630 - val_recall: 0.3342
Epoch 45/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3596 - recall: 0.3699 - val_loss: 0.3604 - val_recall: 0.3464
Epoch 46/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3587 - recall: 0.3550 - val_loss: 0.3573 - val_recall: 0.4005
Epoch 47/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3633 - recall: 0.3827 - val_loss: 0.3618 - val_recall: 0.3538
Epoch 48/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3717 - recall: 0.3571 - val_loss: 0.3646 - val_recall: 0.3440
Epoch 49/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3664 - recall: 0.3352 - val_loss: 0.3766 - val_recall: 0.2678
Epoch 50/50
188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3732 - recall: 0.3182 - val_loss: 0.3646 - val_recall: 0.3047

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history3.history['loss'],adam_dropout_history3.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and showing more noise in both sets

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history3.history['recall'],adam_dropout_history3.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model3.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 961us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model3.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 573us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.348324
Name: Neural Network (64,32) with Adam & Dropout (0.5, 0.5), dtype: float64
recall    0.304668
Name: Neural Network (64,32) with Adam & Dropout (0.5, 0.5), dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.86      0.99      0.92      4777
           1       0.91      0.35      0.50      1223

    accuracy                           0.86      6000
   macro avg       0.88      0.67      0.71      6000
weighted avg       0.87      0.86      0.83      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.85      0.98      0.91      1593
           1       0.83      0.30      0.45       407

    accuracy                           0.85      2000
   macro avg       0.84      0.64      0.68      2000
weighted avg       0.84      0.85      0.82      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• This model is likely too simple to really do much.
• Terrible recall performance.
• Add some layers back infor next model.
• Try He normal to see how it affects a previous model.

Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)"

#Initializing the neural network
adam_he_dropout_model = Sequential()
adam_he_dropout_model.add(Dense(64,activation='relu', kernel_initializer="he_normal", input_dim = X_train.shape[1]))
adam_he_dropout_model.add(Dropout(0.4))
adam_he_dropout_model.add(Dense(32,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.3))
adam_he_dropout_model.add(Dense(16,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.2))
adam_he_dropout_model.add(Dense(8,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.1))
adam_he_dropout_model.add(Dense(1, activation = 'sigmoid', kernel_initializer="he_normal"))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_he_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_he_dropout_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

Loss function

#Fitting the ANN
adam_he_dropout_history = adam_he_dropout_model.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.7711 - recall: 0.4888 - val_loss: 0.4644 - val_recall: 0.0000e+00
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5203 - recall: 0.0572 - val_loss: 0.4496 - val_recall: 0.0000e+00
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4879 - recall: 0.0408 - val_loss: 0.4407 - val_recall: 0.0000e+00
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4721 - recall: 0.0616 - val_loss: 0.4360 - val_recall: 0.0000e+00
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4680 - recall: 0.0574 - val_loss: 0.4321 - val_recall: 0.0000e+00
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4641 - recall: 0.0563 - val_loss: 0.4275 - val_recall: 0.0025
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4593 - recall: 0.0803 - val_loss: 0.4260 - val_recall: 0.0000e+00
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4505 - recall: 0.0973 - val_loss: 0.4241 - val_recall: 0.0098
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4448 - recall: 0.1195 - val_loss: 0.4209 - val_recall: 0.0172
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4479 - recall: 0.1025 - val_loss: 0.4196 - val_recall: 0.0565
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4404 - recall: 0.1329 - val_loss: 0.4137 - val_recall: 0.1376
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4460 - recall: 0.1611 - val_loss: 0.4105 - val_recall: 0.1622
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4384 - recall: 0.1617 - val_loss: 0.4075 - val_recall: 0.1867
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4291 - recall: 0.1844 - val_loss: 0.4045 - val_recall: 0.1990
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4241 - recall: 0.1818 - val_loss: 0.4021 - val_recall: 0.2088
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4246 - recall: 0.2188 - val_loss: 0.3968 - val_recall: 0.2383
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4352 - recall: 0.2378 - val_loss: 0.3954 - val_recall: 0.2334
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4145 - recall: 0.2265 - val_loss: 0.3918 - val_recall: 0.2555
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4244 - recall: 0.2518 - val_loss: 0.3903 - val_recall: 0.2383
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4175 - recall: 0.2456 - val_loss: 0.3862 - val_recall: 0.2850
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4088 - recall: 0.2808 - val_loss: 0.3846 - val_recall: 0.2654
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4045 - recall: 0.2916 - val_loss: 0.3796 - val_recall: 0.3268
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4059 - recall: 0.2760 - val_loss: 0.3784 - val_recall: 0.3120
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4095 - recall: 0.2726 - val_loss: 0.3773 - val_recall: 0.3022
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3934 - recall: 0.3055 - val_loss: 0.3730 - val_recall: 0.3194
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3853 - recall: 0.3311 - val_loss: 0.3707 - val_recall: 0.3194
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3954 - recall: 0.3212 - val_loss: 0.3707 - val_recall: 0.3268
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3906 - recall: 0.3416 - val_loss: 0.3700 - val_recall: 0.3071
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3878 - recall: 0.3264 - val_loss: 0.3670 - val_recall: 0.3268
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3959 - recall: 0.3285 - val_loss: 0.3676 - val_recall: 0.3145
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3835 - recall: 0.3339 - val_loss: 0.3638 - val_recall: 0.3686
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3867 - recall: 0.3409 - val_loss: 0.3643 - val_recall: 0.3464
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3856 - recall: 0.3361 - val_loss: 0.3639 - val_recall: 0.3464
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3774 - recall: 0.3673 - val_loss: 0.3624 - val_recall: 0.3538
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3793 - recall: 0.3680 - val_loss: 0.3606 - val_recall: 0.3661
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3826 - recall: 0.3462 - val_loss: 0.3610 - val_recall: 0.3366
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3730 - recall: 0.3876 - val_loss: 0.3600 - val_recall: 0.3612
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3696 - recall: 0.3866 - val_loss: 0.3594 - val_recall: 0.3686
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3741 - recall: 0.3809 - val_loss: 0.3620 - val_recall: 0.3022
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3677 - recall: 0.3827 - val_loss: 0.3591 - val_recall: 0.3538
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3694 - recall: 0.3876 - val_loss: 0.3597 - val_recall: 0.3415
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3730 - recall: 0.3495 - val_loss: 0.3581 - val_recall: 0.3538
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3736 - recall: 0.3668 - val_loss: 0.3587 - val_recall: 0.3391
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3753 - recall: 0.3758 - val_loss: 0.3579 - val_recall: 0.3464
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3690 - recall: 0.4118 - val_loss: 0.3587 - val_recall: 0.3464
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3737 - recall: 0.3848 - val_loss: 0.3576 - val_recall: 0.3440
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3650 - recall: 0.3883 - val_loss: 0.3575 - val_recall: 0.3440
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3628 - recall: 0.4153 - val_loss: 0.3575 - val_recall: 0.3366
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3566 - recall: 0.3842 - val_loss: 0.3563 - val_recall: 0.3612
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3595 - recall: 0.4283 - val_loss: 0.3576 - val_recall: 0.3317

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_he_dropout_history.history['loss'],adam_he_dropout_history.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and some noise in both sets though much more in training due to dropout

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_he_dropout_history.history['recall'],adam_he_dropout_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_he_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 725us/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_he_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 541us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.376124
Name: Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
recall    0.331695
Name: Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.86      0.99      0.92      4777
           1       0.87      0.38      0.53      1223

    accuracy                           0.86      6000
   macro avg       0.87      0.68      0.72      6000
weighted avg       0.86      0.86      0.84      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.85      0.98      0.91      1593
           1       0.78      0.33      0.47       407

    accuracy                           0.85      2000
   macro avg       0.82      0.65      0.69      2000
weighted avg       0.84      0.85      0.82      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Definitely going the wrong way on these models.
• he normal does not seem to have helped.
• Recall is not acceptable.
• Resimplify and lets try some batch normalization.

Neural Network (64,32,16,8) with Adam & Batch Normalization

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with Adam & Batch Normalization"

#Initializing the neural network
adam_batch_model = Sequential()
adam_batch_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(32,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(16,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(8,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_batch_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_batch_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization             │ (None, 64)             │           256 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_1           │ (None, 32)             │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_2           │ (None, 16)             │            64 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_3           │ (None, 8)              │            32 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 4,001 (15.63 KB)

 Trainable params: 3,761 (14.69 KB)

 Non-trainable params: 240 (960.00 B)

Loss function

#Fitting the ANN with
adam_batch_history = adam_batch_model.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 3s 5ms/step - loss: 0.8402 - recall: 0.6037 - val_loss: 0.5860 - val_recall: 0.2899
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5146 - recall: 0.5745 - val_loss: 0.4742 - val_recall: 0.2752
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4283 - recall: 0.5071 - val_loss: 0.4114 - val_recall: 0.3170
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3754 - recall: 0.4960 - val_loss: 0.3840 - val_recall: 0.3735
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3444 - recall: 0.4964 - val_loss: 0.3733 - val_recall: 0.3931
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3271 - recall: 0.4989 - val_loss: 0.3716 - val_recall: 0.3931
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3126 - recall: 0.5199 - val_loss: 0.3724 - val_recall: 0.3907
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3025 - recall: 0.5286 - val_loss: 0.3734 - val_recall: 0.3931
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2928 - recall: 0.5376 - val_loss: 0.3760 - val_recall: 0.4054
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2840 - recall: 0.5670 - val_loss: 0.3806 - val_recall: 0.4201
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2762 - recall: 0.5836 - val_loss: 0.3837 - val_recall: 0.4300
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2686 - recall: 0.6001 - val_loss: 0.3880 - val_recall: 0.4275
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2615 - recall: 0.6027 - val_loss: 0.3960 - val_recall: 0.4545
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2536 - recall: 0.6203 - val_loss: 0.4032 - val_recall: 0.4423
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2465 - recall: 0.6363 - val_loss: 0.4059 - val_recall: 0.4349
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2376 - recall: 0.6507 - val_loss: 0.4157 - val_recall: 0.4668
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2340 - recall: 0.6598 - val_loss: 0.4234 - val_recall: 0.4398
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2266 - recall: 0.6708 - val_loss: 0.4382 - val_recall: 0.4595
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2215 - recall: 0.6858 - val_loss: 0.4471 - val_recall: 0.4693
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2165 - recall: 0.7009 - val_loss: 0.4567 - val_recall: 0.4644
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2089 - recall: 0.6971 - val_loss: 0.4639 - val_recall: 0.4545
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.2028 - recall: 0.7081 - val_loss: 0.4743 - val_recall: 0.4619
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1990 - recall: 0.7232 - val_loss: 0.4882 - val_recall: 0.4521
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1938 - recall: 0.7346 - val_loss: 0.4949 - val_recall: 0.4472
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1842 - recall: 0.7461 - val_loss: 0.5075 - val_recall: 0.4324
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1801 - recall: 0.7626 - val_loss: 0.5213 - val_recall: 0.4472
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1769 - recall: 0.7582 - val_loss: 0.5318 - val_recall: 0.4398
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1700 - recall: 0.7787 - val_loss: 0.5395 - val_recall: 0.4398
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1638 - recall: 0.7802 - val_loss: 0.5532 - val_recall: 0.4373
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1651 - recall: 0.7861 - val_loss: 0.5591 - val_recall: 0.4373
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1568 - recall: 0.7920 - val_loss: 0.5604 - val_recall: 0.4398
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1547 - recall: 0.8044 - val_loss: 0.5881 - val_recall: 0.4447
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1583 - recall: 0.7999 - val_loss: 0.5882 - val_recall: 0.4472
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1459 - recall: 0.8184 - val_loss: 0.6011 - val_recall: 0.4275
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1393 - recall: 0.8176 - val_loss: 0.6113 - val_recall: 0.4423
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1364 - recall: 0.8239 - val_loss: 0.6347 - val_recall: 0.4398
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1406 - recall: 0.8190 - val_loss: 0.6339 - val_recall: 0.4496
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1350 - recall: 0.8273 - val_loss: 0.6184 - val_recall: 0.4398
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1352 - recall: 0.8240 - val_loss: 0.6592 - val_recall: 0.4373
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1401 - recall: 0.8224 - val_loss: 0.6691 - val_recall: 0.4251
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1246 - recall: 0.8406 - val_loss: 0.6608 - val_recall: 0.4472
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1223 - recall: 0.8458 - val_loss: 0.6857 - val_recall: 0.4300
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1228 - recall: 0.8430 - val_loss: 0.6779 - val_recall: 0.4201
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1132 - recall: 0.8558 - val_loss: 0.7129 - val_recall: 0.4275
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1071 - recall: 0.8676 - val_loss: 0.7156 - val_recall: 0.4079
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1071 - recall: 0.8707 - val_loss: 0.7528 - val_recall: 0.4496
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1025 - recall: 0.8780 - val_loss: 0.7649 - val_recall: 0.4152
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1092 - recall: 0.8712 - val_loss: 0.7722 - val_recall: 0.4275
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1142 - recall: 0.8612 - val_loss: 0.7864 - val_recall: 0.4177
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.1013 - recall: 0.8678 - val_loss: 0.8063 - val_recall: 0.4619

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_batch_history.history['loss'],adam_batch_history.history['val_loss'])

• The loss in the validation and train are drastically divergent after epoch 4 or so. Small evidence of noise in both sets.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_batch_history.history['recall'],adam_batch_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_batch_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_batch_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 623us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.683565
Name: Neural Network (64,32,16,8) with Adam & Batch Normalization, dtype: float64
recall    0.461916
Name: Neural Network (64,32,16,8) with Adam & Batch Normalization, dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.92      0.95      0.93      4777
           1       0.77      0.68      0.72      1223

    accuracy                           0.89      6000
   macro avg       0.85      0.82      0.83      6000
weighted avg       0.89      0.89      0.89      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.87      0.90      0.88      1593
           1       0.54      0.46      0.50       407

    accuracy                           0.81      2000
   macro avg       0.71      0.68      0.69      2000
weighted avg       0.80      0.81      0.81      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Batch Normalization didn't do much to help here, still pretty bad recall performance.
• Lets combine with dropout for the next model.

Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)"

#Initializing the neural network
adam_batch_dropout_model = Sequential()
adam_batch_dropout_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.4))
adam_batch_dropout_model.add(Dense(32,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.3))
adam_batch_dropout_model.add(Dense(16,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.2))
adam_batch_dropout_model.add(Dense(8,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.1))
adam_batch_dropout_model.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

## compile the model with binary cross entropy as loss function and recall as the metric.
adam_batch_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

# Summary of the model
adam_batch_dropout_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization             │ (None, 64)             │           256 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_1           │ (None, 32)             │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_2           │ (None, 16)             │            64 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_3           │ (None, 8)              │            32 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 4,001 (15.63 KB)

 Trainable params: 3,761 (14.69 KB)

 Non-trainable params: 240 (960.00 B)

Loss function

#Fitting the ANN
adam_batch_dropout_history = adam_batch_dropout_model.fit(
    X_train,y_train,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data=(X_val,y_val)
)

Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 3s 5ms/step - loss: 0.7578 - recall: 0.4486 - val_loss: 0.5735 - val_recall: 0.1106
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6124 - recall: 0.4395 - val_loss: 0.5079 - val_recall: 0.1597
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5428 - recall: 0.3831 - val_loss: 0.4586 - val_recall: 0.1892
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4917 - recall: 0.3463 - val_loss: 0.4274 - val_recall: 0.2678
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4627 - recall: 0.3197 - val_loss: 0.4091 - val_recall: 0.2604
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4371 - recall: 0.2924 - val_loss: 0.3958 - val_recall: 0.2875
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4320 - recall: 0.2972 - val_loss: 0.3866 - val_recall: 0.3194
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4202 - recall: 0.3022 - val_loss: 0.3802 - val_recall: 0.2998
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4161 - recall: 0.3153 - val_loss: 0.3772 - val_recall: 0.3120
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4114 - recall: 0.2951 - val_loss: 0.3701 - val_recall: 0.3784
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4091 - recall: 0.3030 - val_loss: 0.3682 - val_recall: 0.3538
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3996 - recall: 0.3228 - val_loss: 0.3654 - val_recall: 0.3268
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3951 - recall: 0.3267 - val_loss: 0.3616 - val_recall: 0.3612
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3903 - recall: 0.3538 - val_loss: 0.3605 - val_recall: 0.3391
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3870 - recall: 0.3623 - val_loss: 0.3592 - val_recall: 0.3514
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3868 - recall: 0.3482 - val_loss: 0.3587 - val_recall: 0.3464
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3875 - recall: 0.3509 - val_loss: 0.3581 - val_recall: 0.3710
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3867 - recall: 0.3550 - val_loss: 0.3595 - val_recall: 0.3612
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3828 - recall: 0.3429 - val_loss: 0.3583 - val_recall: 0.3661
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3796 - recall: 0.3467 - val_loss: 0.3585 - val_recall: 0.3415
Epoch 21/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3765 - recall: 0.3724 - val_loss: 0.3580 - val_recall: 0.3415
Epoch 22/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3686 - recall: 0.3711 - val_loss: 0.3570 - val_recall: 0.3538
Epoch 23/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3770 - recall: 0.3799 - val_loss: 0.3559 - val_recall: 0.3538
Epoch 24/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3740 - recall: 0.3685 - val_loss: 0.3568 - val_recall: 0.3440
Epoch 25/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3677 - recall: 0.3700 - val_loss: 0.3569 - val_recall: 0.3514
Epoch 26/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3636 - recall: 0.3824 - val_loss: 0.3565 - val_recall: 0.3563
Epoch 27/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3646 - recall: 0.4192 - val_loss: 0.3550 - val_recall: 0.3636
Epoch 28/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3580 - recall: 0.4192 - val_loss: 0.3555 - val_recall: 0.3636
Epoch 29/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3649 - recall: 0.3852 - val_loss: 0.3554 - val_recall: 0.3661
Epoch 30/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3653 - recall: 0.4026 - val_loss: 0.3557 - val_recall: 0.3636
Epoch 31/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3574 - recall: 0.4182 - val_loss: 0.3580 - val_recall: 0.3710
Epoch 32/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3565 - recall: 0.4211 - val_loss: 0.3582 - val_recall: 0.3784
Epoch 33/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3681 - recall: 0.3878 - val_loss: 0.3556 - val_recall: 0.3833
Epoch 34/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3581 - recall: 0.4149 - val_loss: 0.3551 - val_recall: 0.3759
Epoch 35/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3592 - recall: 0.3985 - val_loss: 0.3562 - val_recall: 0.3661
Epoch 36/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3643 - recall: 0.3872 - val_loss: 0.3560 - val_recall: 0.3563
Epoch 37/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3626 - recall: 0.4112 - val_loss: 0.3551 - val_recall: 0.3489
Epoch 38/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3572 - recall: 0.4158 - val_loss: 0.3555 - val_recall: 0.3636
Epoch 39/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3573 - recall: 0.4160 - val_loss: 0.3554 - val_recall: 0.3735
Epoch 40/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3530 - recall: 0.4301 - val_loss: 0.3552 - val_recall: 0.3808
Epoch 41/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3611 - recall: 0.3974 - val_loss: 0.3567 - val_recall: 0.3735
Epoch 42/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3608 - recall: 0.4059 - val_loss: 0.3588 - val_recall: 0.3612
Epoch 43/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3578 - recall: 0.4137 - val_loss: 0.3575 - val_recall: 0.3587
Epoch 44/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3538 - recall: 0.4136 - val_loss: 0.3549 - val_recall: 0.3956
Epoch 45/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3584 - recall: 0.4348 - val_loss: 0.3558 - val_recall: 0.3735
Epoch 46/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3490 - recall: 0.4300 - val_loss: 0.3580 - val_recall: 0.3686
Epoch 47/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3481 - recall: 0.4219 - val_loss: 0.3572 - val_recall: 0.3808
Epoch 48/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3531 - recall: 0.4404 - val_loss: 0.3566 - val_recall: 0.3759
Epoch 49/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3543 - recall: 0.4281 - val_loss: 0.3575 - val_recall: 0.3931
Epoch 50/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.3560 - recall: 0.4223 - val_loss: 0.3580 - val_recall: 0.3661

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_batch_dropout_history.history['loss'],adam_batch_dropout_history.history['val_loss'])

• The loss in the validation and train are similar, and begin converging around epoch 38, and some noise in both sets though much more in training due to dropout

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_batch_dropout_history.history['recall'],adam_batch_dropout_history.history['val_recall'])

#Predicting the results using best as a threshold
y_train_pred = adam_batch_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred

188/188 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step

array([[False],
       [False],
       [False],
       ...,
       [ True],
       [False],
       [ True]])

#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_batch_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 617us/step

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.413737
Name: Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
recall    0.366093
Name: Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64

Classification report

#classification report
print(classification_report(y_train,y_train_pred))

              precision    recall  f1-score   support

           0       0.87      0.98      0.92      4777
           1       0.84      0.41      0.55      1223

    accuracy                           0.86      6000
   macro avg       0.85      0.70      0.74      6000
weighted avg       0.86      0.86      0.85      6000

#classification report
print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.86      0.98      0.91      1593
           1       0.81      0.37      0.50       407

    accuracy                           0.85      2000
   macro avg       0.83      0.67      0.71      2000
weighted avg       0.85      0.85      0.83      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Even worse performance. In this case I do not think batch normalization should be used at all.
• Next model we will retry SGD, but use SMOTE to have better data to train on.

Neural Network (64,32,16,8) with SMOTE & SGD Optimizer

sm  = SMOTE(random_state=42)
# fit SMOTE on the training data.
X_train_smote, y_train_smote= sm.fit_resample(X_train,y_train)
print('After UpSampling, the shape of train_X: {}'.format(X_train_smote.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_smote.shape))

After UpSampling, the shape of train_X: (9554, 11)
After UpSampling, the shape of train_y: (9554,) 

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE & SGD"

#Initializing the model
sgd_smote_model = Sequential()
sgd_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
sgd_smote_model.add(Dense(32,activation='relu'))
sgd_smote_model.add(Dense(16,activation='relu'))
sgd_smote_model.add(Dense(8,activation='relu'))
sgd_smote_model.add(Dense(1, activation = 'sigmoid'))

# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
sgd_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

sgd_smote_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

#Fitting the ANN
sgd_smote_history = sgd_smote_model.fit(
    X_train_smote, y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    #class_weight=cw_dict,
    verbose=1,
    validation_data = (X_val,y_val)
)

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6940 - recall: 0.3086 - val_loss: 0.6787 - val_recall: 0.5332
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 984us/step - loss: 0.6782 - recall: 0.5645 - val_loss: 0.6559 - val_recall: 0.6118
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 998us/step - loss: 0.6551 - recall: 0.5830 - val_loss: 0.6205 - val_recall: 0.6658
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 991us/step - loss: 0.6234 - recall: 0.6380 - val_loss: 0.5968 - val_recall: 0.7174
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 981us/step - loss: 0.5951 - recall: 0.6714 - val_loss: 0.5836 - val_recall: 0.7273
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5763 - recall: 0.6981 - val_loss: 0.5781 - val_recall: 0.7322
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 986us/step - loss: 0.5652 - recall: 0.7168 - val_loss: 0.5760 - val_recall: 0.7396
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5571 - recall: 0.7283 - val_loss: 0.5725 - val_recall: 0.7445
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5495 - recall: 0.7338 - val_loss: 0.5671 - val_recall: 0.7469
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5418 - recall: 0.7401 - val_loss: 0.5644 - val_recall: 0.7469
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 990us/step - loss: 0.5345 - recall: 0.7445 - val_loss: 0.5614 - val_recall: 0.7543
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 970us/step - loss: 0.5278 - recall: 0.7493 - val_loss: 0.5578 - val_recall: 0.7543
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 979us/step - loss: 0.5213 - recall: 0.7544 - val_loss: 0.5546 - val_recall: 0.7518
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5149 - recall: 0.7547 - val_loss: 0.5505 - val_recall: 0.7617
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 980us/step - loss: 0.5085 - recall: 0.7595 - val_loss: 0.5475 - val_recall: 0.7641
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 984us/step - loss: 0.5023 - recall: 0.7630 - val_loss: 0.5415 - val_recall: 0.7690
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 962us/step - loss: 0.4959 - recall: 0.7669 - val_loss: 0.5362 - val_recall: 0.7690
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 968us/step - loss: 0.4898 - recall: 0.7704 - val_loss: 0.5300 - val_recall: 0.7715
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 981us/step - loss: 0.4837 - recall: 0.7720 - val_loss: 0.5239 - val_recall: 0.7764
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 986us/step - loss: 0.4780 - recall: 0.7706 - val_loss: 0.5193 - val_recall: 0.7813
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 994us/step - loss: 0.4728 - recall: 0.7695 - val_loss: 0.5132 - val_recall: 0.7764
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 988us/step - loss: 0.4680 - recall: 0.7711 - val_loss: 0.5075 - val_recall: 0.7764
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4638 - recall: 0.7704 - val_loss: 0.5031 - val_recall: 0.7715
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4600 - recall: 0.7698 - val_loss: 0.4988 - val_recall: 0.7715
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 963us/step - loss: 0.4566 - recall: 0.7729 - val_loss: 0.4953 - val_recall: 0.7715
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 969us/step - loss: 0.4534 - recall: 0.7737 - val_loss: 0.4919 - val_recall: 0.7740
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 974us/step - loss: 0.4506 - recall: 0.7738 - val_loss: 0.4888 - val_recall: 0.7715
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 962us/step - loss: 0.4480 - recall: 0.7743 - val_loss: 0.4856 - val_recall: 0.7641
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 981us/step - loss: 0.4456 - recall: 0.7751 - val_loss: 0.4825 - val_recall: 0.7592
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4432 - recall: 0.7744 - val_loss: 0.4805 - val_recall: 0.7592
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4410 - recall: 0.7746 - val_loss: 0.4785 - val_recall: 0.7592
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 990us/step - loss: 0.4388 - recall: 0.7755 - val_loss: 0.4756 - val_recall: 0.7518
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 988us/step - loss: 0.4367 - recall: 0.7776 - val_loss: 0.4732 - val_recall: 0.7494
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4347 - recall: 0.7780 - val_loss: 0.4716 - val_recall: 0.7494
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4327 - recall: 0.7794 - val_loss: 0.4697 - val_recall: 0.7494
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4307 - recall: 0.7800 - val_loss: 0.4683 - val_recall: 0.7494
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4289 - recall: 0.7819 - val_loss: 0.4665 - val_recall: 0.7469
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 982us/step - loss: 0.4271 - recall: 0.7830 - val_loss: 0.4650 - val_recall: 0.7445
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 970us/step - loss: 0.4255 - recall: 0.7840 - val_loss: 0.4642 - val_recall: 0.7469
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4240 - recall: 0.7839 - val_loss: 0.4632 - val_recall: 0.7445
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4225 - recall: 0.7858 - val_loss: 0.4626 - val_recall: 0.7420
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 995us/step - loss: 0.4211 - recall: 0.7858 - val_loss: 0.4620 - val_recall: 0.7396
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 983us/step - loss: 0.4197 - recall: 0.7861 - val_loss: 0.4606 - val_recall: 0.7396
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 990us/step - loss: 0.4183 - recall: 0.7877 - val_loss: 0.4592 - val_recall: 0.7396
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 980us/step - loss: 0.4170 - recall: 0.7872 - val_loss: 0.4582 - val_recall: 0.7322
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 980us/step - loss: 0.4158 - recall: 0.7891 - val_loss: 0.4569 - val_recall: 0.7248
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4145 - recall: 0.7878 - val_loss: 0.4560 - val_recall: 0.7248
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4133 - recall: 0.7875 - val_loss: 0.4555 - val_recall: 0.7248
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 982us/step - loss: 0.4121 - recall: 0.7913 - val_loss: 0.4550 - val_recall: 0.7248
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 979us/step - loss: 0.4110 - recall: 0.7929 - val_loss: 0.4550 - val_recall: 0.7273

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(sgd_smote_history.history['loss'],sgd_smote_history.history['val_loss'])

• The loss in the validation and train are similar, though divergent, with less noise evident.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_smote_history.history['recall'],sgd_smote_history.history['val_recall'])

y_train_pred = sgd_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 647us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = sgd_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 559us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.805526
Name: Neural Network (64,32,16,8) with SMOTE & SGD, dtype: float64
recall    0.727273
Name: Neural Network (64,32,16,8) with SMOTE & SGD, dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.81      0.82      0.81      4777
           1       0.82      0.81      0.81      4777

    accuracy                           0.81      9554
   macro avg       0.81      0.81      0.81      9554
weighted avg       0.81      0.81      0.81      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.92      0.80      0.86      1593
           1       0.49      0.73      0.58       407

    accuracy                           0.79      2000
   macro avg       0.70      0.77      0.72      2000
weighted avg       0.83      0.79      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Recall scores improving.
• Lets try with Adam.

Neural Network (64,32,16,8) with SMOTE & Adam Optimizer

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE & Adam"

#Initializing the model
adam_smote_model = Sequential()
adam_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_smote_model.add(Dense(32,activation='relu'))
adam_smote_model.add(Dense(16,activation='relu'))
adam_smote_model.add(Dense(8,activation='relu'))
adam_smote_model.add(Dense(1, activation = 'sigmoid'))

adam_smote_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_smote_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

#Fitting the ANN

adam_smote_history = adam_smote_model.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val)
)

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6328 - recall: 0.5413 - val_loss: 0.5316 - val_recall: 0.7346
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5091 - recall: 0.7609 - val_loss: 0.4842 - val_recall: 0.7543
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4638 - recall: 0.7730 - val_loss: 0.4608 - val_recall: 0.7346
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4445 - recall: 0.7807 - val_loss: 0.4580 - val_recall: 0.7224
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4318 - recall: 0.7911 - val_loss: 0.4555 - val_recall: 0.7076
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4211 - recall: 0.7970 - val_loss: 0.4546 - val_recall: 0.7150
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4126 - recall: 0.8052 - val_loss: 0.4568 - val_recall: 0.7125
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4043 - recall: 0.8111 - val_loss: 0.4567 - val_recall: 0.7125
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3971 - recall: 0.8144 - val_loss: 0.4569 - val_recall: 0.7002
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3902 - recall: 0.8181 - val_loss: 0.4597 - val_recall: 0.7027
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3835 - recall: 0.8271 - val_loss: 0.4669 - val_recall: 0.7101
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3769 - recall: 0.8310 - val_loss: 0.4701 - val_recall: 0.7076
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3708 - recall: 0.8386 - val_loss: 0.4712 - val_recall: 0.7125
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3647 - recall: 0.8414 - val_loss: 0.4743 - val_recall: 0.7052
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3583 - recall: 0.8468 - val_loss: 0.4742 - val_recall: 0.6978
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3526 - recall: 0.8488 - val_loss: 0.4796 - val_recall: 0.6953
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3463 - recall: 0.8552 - val_loss: 0.4841 - val_recall: 0.6929
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3409 - recall: 0.8573 - val_loss: 0.4881 - val_recall: 0.6880
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3367 - recall: 0.8610 - val_loss: 0.4962 - val_recall: 0.6929
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3314 - recall: 0.8665 - val_loss: 0.4940 - val_recall: 0.6929
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3270 - recall: 0.8689 - val_loss: 0.4986 - val_recall: 0.6880
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3230 - recall: 0.8742 - val_loss: 0.5001 - val_recall: 0.6904
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3177 - recall: 0.8742 - val_loss: 0.5062 - val_recall: 0.6929
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3136 - recall: 0.8789 - val_loss: 0.5107 - val_recall: 0.6929
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3094 - recall: 0.8854 - val_loss: 0.5137 - val_recall: 0.6904
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3049 - recall: 0.8883 - val_loss: 0.5238 - val_recall: 0.6953
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.3003 - recall: 0.8889 - val_loss: 0.5297 - val_recall: 0.6855
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2966 - recall: 0.8918 - val_loss: 0.5304 - val_recall: 0.6806
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2926 - recall: 0.8939 - val_loss: 0.5325 - val_recall: 0.6732
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2888 - recall: 0.8954 - val_loss: 0.5390 - val_recall: 0.6732
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2855 - recall: 0.8988 - val_loss: 0.5455 - val_recall: 0.6683
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2820 - recall: 0.9005 - val_loss: 0.5497 - val_recall: 0.6708
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2797 - recall: 0.9014 - val_loss: 0.5564 - val_recall: 0.6683
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2758 - recall: 0.9054 - val_loss: 0.5634 - val_recall: 0.6732
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2736 - recall: 0.9070 - val_loss: 0.5692 - val_recall: 0.6732
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2710 - recall: 0.9087 - val_loss: 0.5763 - val_recall: 0.6757
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2685 - recall: 0.9096 - val_loss: 0.5781 - val_recall: 0.6708
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2655 - recall: 0.9126 - val_loss: 0.5816 - val_recall: 0.6732
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2622 - recall: 0.9152 - val_loss: 0.5890 - val_recall: 0.6732
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2588 - recall: 0.9166 - val_loss: 0.5959 - val_recall: 0.6683
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2565 - recall: 0.9180 - val_loss: 0.6005 - val_recall: 0.6560
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2545 - recall: 0.9176 - val_loss: 0.6070 - val_recall: 0.6634
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2520 - recall: 0.9182 - val_loss: 0.6118 - val_recall: 0.6536
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2492 - recall: 0.9162 - val_loss: 0.6176 - val_recall: 0.6560
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2471 - recall: 0.9169 - val_loss: 0.6258 - val_recall: 0.6634
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2441 - recall: 0.9177 - val_loss: 0.6342 - val_recall: 0.6634
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2422 - recall: 0.9195 - val_loss: 0.6396 - val_recall: 0.6585
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2410 - recall: 0.9201 - val_loss: 0.6448 - val_recall: 0.6560
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2389 - recall: 0.9234 - val_loss: 0.6539 - val_recall: 0.6560
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.2362 - recall: 0.9260 - val_loss: 0.6539 - val_recall: 0.6511

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_smote_history.history['loss'],adam_smote_history.history['val_loss'])

• The loss in the validation and train diverge after the first couple of epochs. Small amount of noise evident in validation, while training is smooth.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_smote_history.history['recall'],adam_smote_history.history['val_recall'])

y_train_pred = adam_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 660us/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 550us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [False]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.941386
Name: Neural Network (64,32,16,8) with SMOTE & Adam, dtype: float64
recall    0.651106
Name: Neural Network (64,32,16,8) with SMOTE & Adam, dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.94      0.88      0.91      4777
           1       0.88      0.94      0.91      4777

    accuracy                           0.91      9554
   macro avg       0.91      0.91      0.91      9554
weighted avg       0.91      0.91      0.91      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.90      0.82      0.86      1593
           1       0.48      0.65      0.56       407

    accuracy                           0.79      2000
   macro avg       0.69      0.74      0.71      2000
weighted avg       0.82      0.79      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Good performance on training set recall, but vlaidation is going the wrong way.
• Lets see if adding dropout back in helps.

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1)"

#Initializing the model
adam_dropout_smote_model001 = Sequential()
adam_dropout_smote_model001.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model001.add(Dropout(0.8))
adam_dropout_smote_model001.add(Dense(32,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.4))
adam_dropout_smote_model001.add(Dense(16,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.2))
adam_dropout_smote_model001.add(Dense(8,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.1))
adam_dropout_smote_model001.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model001.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model001.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history001 = adam_dropout_smote_model001.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.6936 - recall: 0.5652 - val_loss: 0.6665 - val_recall: 0.4373
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6737 - recall: 0.5789 - val_loss: 0.6285 - val_recall: 0.6658
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6447 - recall: 0.6325 - val_loss: 0.5874 - val_recall: 0.6757
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6238 - recall: 0.6817 - val_loss: 0.5840 - val_recall: 0.6929
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6102 - recall: 0.7034 - val_loss: 0.5659 - val_recall: 0.6806
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5994 - recall: 0.7122 - val_loss: 0.5747 - val_recall: 0.6929
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5941 - recall: 0.7199 - val_loss: 0.5692 - val_recall: 0.7199
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5940 - recall: 0.7292 - val_loss: 0.5688 - val_recall: 0.7322
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5749 - recall: 0.7200 - val_loss: 0.5651 - val_recall: 0.7322
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5747 - recall: 0.7319 - val_loss: 0.5509 - val_recall: 0.7248
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5621 - recall: 0.7398 - val_loss: 0.5479 - val_recall: 0.7273
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5668 - recall: 0.7204 - val_loss: 0.5484 - val_recall: 0.7224
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5552 - recall: 0.7190 - val_loss: 0.5431 - val_recall: 0.7445
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5545 - recall: 0.7339 - val_loss: 0.5368 - val_recall: 0.7396
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5509 - recall: 0.7205 - val_loss: 0.5186 - val_recall: 0.7420
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5478 - recall: 0.7267 - val_loss: 0.5284 - val_recall: 0.7641
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5384 - recall: 0.7392 - val_loss: 0.5158 - val_recall: 0.7690
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5382 - recall: 0.7329 - val_loss: 0.5072 - val_recall: 0.7543
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5350 - recall: 0.7327 - val_loss: 0.5163 - val_recall: 0.7764
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5301 - recall: 0.7459 - val_loss: 0.5057 - val_recall: 0.7518
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5264 - recall: 0.7420 - val_loss: 0.5016 - val_recall: 0.7789
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5235 - recall: 0.7550 - val_loss: 0.5136 - val_recall: 0.7887
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5246 - recall: 0.7442 - val_loss: 0.4978 - val_recall: 0.7690
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5181 - recall: 0.7430 - val_loss: 0.4960 - val_recall: 0.7666
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5240 - recall: 0.7515 - val_loss: 0.5105 - val_recall: 0.8059
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5139 - recall: 0.7613 - val_loss: 0.5018 - val_recall: 0.8108
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5099 - recall: 0.7471 - val_loss: 0.5035 - val_recall: 0.7936
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5132 - recall: 0.7586 - val_loss: 0.5054 - val_recall: 0.8059
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5138 - recall: 0.7538 - val_loss: 0.5024 - val_recall: 0.7912
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5065 - recall: 0.7623 - val_loss: 0.4942 - val_recall: 0.7764
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5091 - recall: 0.7516 - val_loss: 0.4914 - val_recall: 0.7592
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5091 - recall: 0.7583 - val_loss: 0.4863 - val_recall: 0.7789
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5043 - recall: 0.7657 - val_loss: 0.4869 - val_recall: 0.7617
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4994 - recall: 0.7532 - val_loss: 0.4919 - val_recall: 0.7592
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5031 - recall: 0.7582 - val_loss: 0.4884 - val_recall: 0.7813
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5006 - recall: 0.7559 - val_loss: 0.4807 - val_recall: 0.7617
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4987 - recall: 0.7628 - val_loss: 0.4800 - val_recall: 0.7985
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5025 - recall: 0.7629 - val_loss: 0.4768 - val_recall: 0.7862
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4932 - recall: 0.7589 - val_loss: 0.4843 - val_recall: 0.7592
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5006 - recall: 0.7640 - val_loss: 0.4771 - val_recall: 0.7445
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4965 - recall: 0.7684 - val_loss: 0.4798 - val_recall: 0.8059
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4973 - recall: 0.7644 - val_loss: 0.4882 - val_recall: 0.8034
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4867 - recall: 0.7713 - val_loss: 0.4802 - val_recall: 0.7543
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5004 - recall: 0.7565 - val_loss: 0.4834 - val_recall: 0.7469
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4857 - recall: 0.7607 - val_loss: 0.4721 - val_recall: 0.7445
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4868 - recall: 0.7642 - val_loss: 0.4810 - val_recall: 0.7494
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7591 - val_loss: 0.4828 - val_recall: 0.7445
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4950 - recall: 0.7679 - val_loss: 0.4767 - val_recall: 0.7322
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4900 - recall: 0.7549 - val_loss: 0.4817 - val_recall: 0.7346
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4887 - recall: 0.7631 - val_loss: 0.4792 - val_recall: 0.7297

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history001.history['loss'],adam_dropout_smote_history001.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history001.history['recall'],adam_dropout_smote_history001.history['val_recall'])

y_train_pred = adam_dropout_smote_model001.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 659us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model001.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 565us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.793804
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1), dtype: float64
recall    0.72973
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.80      0.81      0.80      4777
           1       0.81      0.79      0.80      4777

    accuracy                           0.80      9554
   macro avg       0.80      0.80      0.80      9554
weighted avg       0.80      0.80      0.80      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.92      0.80      0.86      1593
           1       0.48      0.73      0.58       407

    accuracy                           0.78      2000
   macro avg       0.70      0.76      0.72      2000
weighted avg       0.83      0.78      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Dropout seems to have helped improve recall scores.
• Lets try adjusting the Learning rate. We'll start with 0.001

Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)"

#Initializing the model
adam_dropout_smote_model001 = Sequential()
adam_dropout_smote_model001.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model001.add(Dropout(0.8))
adam_dropout_smote_model001.add(Dense(32,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.4))
adam_dropout_smote_model001.add(Dense(16,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.2))
adam_dropout_smote_model001.add(Dense(8,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.1))
adam_dropout_smote_model001.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model001.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model001.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history001 = adam_dropout_smote_model001.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.6936 - recall: 0.5652 - val_loss: 0.6665 - val_recall: 0.4373
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6737 - recall: 0.5789 - val_loss: 0.6285 - val_recall: 0.6658
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6447 - recall: 0.6325 - val_loss: 0.5874 - val_recall: 0.6757
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6238 - recall: 0.6817 - val_loss: 0.5840 - val_recall: 0.6929
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6102 - recall: 0.7034 - val_loss: 0.5659 - val_recall: 0.6806
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5994 - recall: 0.7122 - val_loss: 0.5747 - val_recall: 0.6929
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5941 - recall: 0.7199 - val_loss: 0.5692 - val_recall: 0.7199
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5940 - recall: 0.7292 - val_loss: 0.5688 - val_recall: 0.7322
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5749 - recall: 0.7200 - val_loss: 0.5651 - val_recall: 0.7322
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5747 - recall: 0.7319 - val_loss: 0.5509 - val_recall: 0.7248
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5621 - recall: 0.7398 - val_loss: 0.5479 - val_recall: 0.7273
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5668 - recall: 0.7204 - val_loss: 0.5484 - val_recall: 0.7224
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5552 - recall: 0.7190 - val_loss: 0.5431 - val_recall: 0.7445
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5545 - recall: 0.7339 - val_loss: 0.5368 - val_recall: 0.7396
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5509 - recall: 0.7205 - val_loss: 0.5186 - val_recall: 0.7420
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5478 - recall: 0.7267 - val_loss: 0.5284 - val_recall: 0.7641
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5384 - recall: 0.7392 - val_loss: 0.5158 - val_recall: 0.7690
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5382 - recall: 0.7329 - val_loss: 0.5072 - val_recall: 0.7543
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5350 - recall: 0.7327 - val_loss: 0.5163 - val_recall: 0.7764
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5301 - recall: 0.7459 - val_loss: 0.5057 - val_recall: 0.7518
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5264 - recall: 0.7420 - val_loss: 0.5016 - val_recall: 0.7789
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5235 - recall: 0.7550 - val_loss: 0.5136 - val_recall: 0.7887
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5246 - recall: 0.7442 - val_loss: 0.4978 - val_recall: 0.7690
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5181 - recall: 0.7430 - val_loss: 0.4960 - val_recall: 0.7666
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5240 - recall: 0.7515 - val_loss: 0.5105 - val_recall: 0.8059
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5139 - recall: 0.7613 - val_loss: 0.5018 - val_recall: 0.8108
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5099 - recall: 0.7471 - val_loss: 0.5035 - val_recall: 0.7936
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5132 - recall: 0.7586 - val_loss: 0.5054 - val_recall: 0.8059
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5138 - recall: 0.7538 - val_loss: 0.5024 - val_recall: 0.7912
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5065 - recall: 0.7623 - val_loss: 0.4942 - val_recall: 0.7764
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5091 - recall: 0.7516 - val_loss: 0.4914 - val_recall: 0.7592
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5091 - recall: 0.7583 - val_loss: 0.4863 - val_recall: 0.7789
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5043 - recall: 0.7657 - val_loss: 0.4869 - val_recall: 0.7617
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4994 - recall: 0.7532 - val_loss: 0.4919 - val_recall: 0.7592
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5031 - recall: 0.7582 - val_loss: 0.4884 - val_recall: 0.7813
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5006 - recall: 0.7559 - val_loss: 0.4807 - val_recall: 0.7617
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4987 - recall: 0.7628 - val_loss: 0.4800 - val_recall: 0.7985
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5025 - recall: 0.7629 - val_loss: 0.4768 - val_recall: 0.7862
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4932 - recall: 0.7589 - val_loss: 0.4843 - val_recall: 0.7592
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5006 - recall: 0.7640 - val_loss: 0.4771 - val_recall: 0.7445
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4965 - recall: 0.7684 - val_loss: 0.4798 - val_recall: 0.8059
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4973 - recall: 0.7644 - val_loss: 0.4882 - val_recall: 0.8034
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4867 - recall: 0.7713 - val_loss: 0.4802 - val_recall: 0.7543
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5004 - recall: 0.7565 - val_loss: 0.4834 - val_recall: 0.7469
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4857 - recall: 0.7607 - val_loss: 0.4721 - val_recall: 0.7445
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4868 - recall: 0.7642 - val_loss: 0.4810 - val_recall: 0.7494
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7591 - val_loss: 0.4828 - val_recall: 0.7445
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4950 - recall: 0.7679 - val_loss: 0.4767 - val_recall: 0.7322
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4900 - recall: 0.7549 - val_loss: 0.4817 - val_recall: 0.7346
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4887 - recall: 0.7631 - val_loss: 0.4792 - val_recall: 0.7297

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history001.history['loss'],adam_dropout_smote_history001.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history001.history['recall'],adam_dropout_smote_history001.history['val_recall'])

y_train_pred = adam_dropout_smote_model001.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model001.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 547us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.793804
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001), dtype: float64
recall    0.72973
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.80      0.81      0.80      4777
           1       0.81      0.79      0.80      4777

    accuracy                           0.80      9554
   macro avg       0.80      0.80      0.80      9554
weighted avg       0.80      0.80      0.80      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.92      0.80      0.86      1593
           1       0.48      0.73      0.58       407

    accuracy                           0.78      2000
   macro avg       0.70      0.76      0.72      2000
weighted avg       0.83      0.78      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• LR=0.001 didnt change the scores, lets try 0.002.

Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)"

#Initializing the model
adam_dropout_smote_model002 = Sequential()
adam_dropout_smote_model002.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model002.add(Dropout(0.8))
adam_dropout_smote_model002.add(Dense(32,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.4))
adam_dropout_smote_model002.add(Dense(16,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.2))
adam_dropout_smote_model002.add(Dense(8,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.1))
adam_dropout_smote_model002.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.002)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

#  compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model002.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model002.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history002 = adam_dropout_smote_model002.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.6883 - recall: 0.5466 - val_loss: 0.6423 - val_recall: 0.6929
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6478 - recall: 0.6209 - val_loss: 0.6048 - val_recall: 0.7248
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6203 - recall: 0.6821 - val_loss: 0.5800 - val_recall: 0.7076
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5970 - recall: 0.7129 - val_loss: 0.5853 - val_recall: 0.7396
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5849 - recall: 0.7293 - val_loss: 0.5653 - val_recall: 0.7469
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5741 - recall: 0.7282 - val_loss: 0.5781 - val_recall: 0.7690
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5669 - recall: 0.7402 - val_loss: 0.5644 - val_recall: 0.7617
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5667 - recall: 0.7326 - val_loss: 0.5533 - val_recall: 0.7715
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5456 - recall: 0.7130 - val_loss: 0.5625 - val_recall: 0.7887
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5429 - recall: 0.7310 - val_loss: 0.5244 - val_recall: 0.7764
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5350 - recall: 0.7404 - val_loss: 0.5222 - val_recall: 0.7887
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5297 - recall: 0.7494 - val_loss: 0.5327 - val_recall: 0.8059
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5238 - recall: 0.7410 - val_loss: 0.5181 - val_recall: 0.7936
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5245 - recall: 0.7411 - val_loss: 0.5160 - val_recall: 0.7936
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5249 - recall: 0.7478 - val_loss: 0.5066 - val_recall: 0.8182
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5231 - recall: 0.7499 - val_loss: 0.5050 - val_recall: 0.8084
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5137 - recall: 0.7429 - val_loss: 0.5107 - val_recall: 0.8280
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5197 - recall: 0.7403 - val_loss: 0.4949 - val_recall: 0.8108
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5156 - recall: 0.7472 - val_loss: 0.5125 - val_recall: 0.8428
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5100 - recall: 0.7709 - val_loss: 0.5149 - val_recall: 0.8182
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5078 - recall: 0.7429 - val_loss: 0.5067 - val_recall: 0.8256
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5057 - recall: 0.7483 - val_loss: 0.5061 - val_recall: 0.7715
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5097 - recall: 0.7399 - val_loss: 0.5030 - val_recall: 0.8182
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5058 - recall: 0.7553 - val_loss: 0.5007 - val_recall: 0.8182
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5080 - recall: 0.7596 - val_loss: 0.5038 - val_recall: 0.8378
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4994 - recall: 0.7783 - val_loss: 0.5175 - val_recall: 0.8403
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4980 - recall: 0.7580 - val_loss: 0.5124 - val_recall: 0.8256
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5044 - recall: 0.7533 - val_loss: 0.4929 - val_recall: 0.7273
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4958 - recall: 0.7649 - val_loss: 0.4925 - val_recall: 0.8256
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4944 - recall: 0.7567 - val_loss: 0.4926 - val_recall: 0.8305
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5021 - recall: 0.7611 - val_loss: 0.4967 - val_recall: 0.7224
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4959 - recall: 0.7579 - val_loss: 0.4917 - val_recall: 0.8428
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4973 - recall: 0.7707 - val_loss: 0.4785 - val_recall: 0.7101
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4948 - recall: 0.7364 - val_loss: 0.4871 - val_recall: 0.8305
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4995 - recall: 0.7599 - val_loss: 0.4933 - val_recall: 0.8256
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4956 - recall: 0.7523 - val_loss: 0.4797 - val_recall: 0.7346
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4881 - recall: 0.7717 - val_loss: 0.4985 - val_recall: 0.8673
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4978 - recall: 0.7716 - val_loss: 0.4801 - val_recall: 0.8256
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4894 - recall: 0.7636 - val_loss: 0.4834 - val_recall: 0.7322
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4943 - recall: 0.7559 - val_loss: 0.4672 - val_recall: 0.7248
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7495 - val_loss: 0.4885 - val_recall: 0.7445
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4926 - recall: 0.7648 - val_loss: 0.4869 - val_recall: 0.7174
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4857 - recall: 0.7455 - val_loss: 0.4805 - val_recall: 0.7224
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4978 - recall: 0.7312 - val_loss: 0.4924 - val_recall: 0.7396
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4850 - recall: 0.7508 - val_loss: 0.4687 - val_recall: 0.6658
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4858 - recall: 0.7226 - val_loss: 0.4791 - val_recall: 0.7199
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4978 - recall: 0.7585 - val_loss: 0.4862 - val_recall: 0.7101
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4944 - recall: 0.7444 - val_loss: 0.4851 - val_recall: 0.7543
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4913 - recall: 0.7448 - val_loss: 0.4807 - val_recall: 0.7371
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4861 - recall: 0.7558 - val_loss: 0.4793 - val_recall: 0.7248

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history002.history['loss'],adam_dropout_smote_history002.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history002.history['recall'],adam_dropout_smote_history002.history['val_recall'])

y_train_pred = adam_dropout_smote_model002.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 659us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model002.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 573us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.789617
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002), dtype: float64
recall    0.724816
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.79      0.81      0.80      4777
           1       0.81      0.79      0.80      4777

    accuracy                           0.80      9554
   macro avg       0.80      0.80      0.80      9554
weighted avg       0.80      0.80      0.80      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.92      0.80      0.86      1593
           1       0.48      0.72      0.58       407

    accuracy                           0.79      2000
   macro avg       0.70      0.76      0.72      2000
weighted avg       0.83      0.79      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• 0.002 dropped the score a small amount.
• Lets try an order of magnitude shift.

Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)"

#Initializing the model
adam_dropout_smote_model01 = Sequential()
adam_dropout_smote_model01.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model01.add(Dropout(0.8))
adam_dropout_smote_model01.add(Dense(32,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.4))
adam_dropout_smote_model01.add(Dense(16,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.2))
adam_dropout_smote_model01.add(Dense(8,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.1))
adam_dropout_smote_model01.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model01.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model01.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history01 = adam_dropout_smote_model01.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.6688 - recall: 0.5720 - val_loss: 0.5680 - val_recall: 0.7420
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5985 - recall: 0.7138 - val_loss: 0.5430 - val_recall: 0.7469
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5860 - recall: 0.7117 - val_loss: 0.5870 - val_recall: 0.8231
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5708 - recall: 0.7045 - val_loss: 0.5388 - val_recall: 0.7641
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5570 - recall: 0.7177 - val_loss: 0.5326 - val_recall: 0.7838
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5527 - recall: 0.7159 - val_loss: 0.5367 - val_recall: 0.8354
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5464 - recall: 0.7088 - val_loss: 0.5327 - val_recall: 0.7985
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5472 - recall: 0.7244 - val_loss: 0.5078 - val_recall: 0.8034
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5296 - recall: 0.7092 - val_loss: 0.4864 - val_recall: 0.8034
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5339 - recall: 0.7199 - val_loss: 0.5272 - val_recall: 0.8231
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5272 - recall: 0.7370 - val_loss: 0.5153 - val_recall: 0.8084
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5347 - recall: 0.7104 - val_loss: 0.4895 - val_recall: 0.7961
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5299 - recall: 0.7199 - val_loss: 0.5184 - val_recall: 0.7912
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5236 - recall: 0.7191 - val_loss: 0.4812 - val_recall: 0.6929
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5284 - recall: 0.6940 - val_loss: 0.5269 - val_recall: 0.8575
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5333 - recall: 0.7324 - val_loss: 0.5162 - val_recall: 0.8600
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5207 - recall: 0.7369 - val_loss: 0.4998 - val_recall: 0.8329
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5202 - recall: 0.7340 - val_loss: 0.5021 - val_recall: 0.8452
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5247 - recall: 0.7372 - val_loss: 0.5185 - val_recall: 0.8526
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5203 - recall: 0.7481 - val_loss: 0.5085 - val_recall: 0.8084
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5207 - recall: 0.7179 - val_loss: 0.5017 - val_recall: 0.8133
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5200 - recall: 0.7123 - val_loss: 0.5096 - val_recall: 0.8157
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5161 - recall: 0.7263 - val_loss: 0.5368 - val_recall: 0.8649
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5159 - recall: 0.7363 - val_loss: 0.5174 - val_recall: 0.8771
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5220 - recall: 0.7474 - val_loss: 0.5190 - val_recall: 0.8673
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5162 - recall: 0.7433 - val_loss: 0.4944 - val_recall: 0.8575
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5206 - recall: 0.7304 - val_loss: 0.5188 - val_recall: 0.8526
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5177 - recall: 0.7389 - val_loss: 0.5136 - val_recall: 0.8526
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5200 - recall: 0.7538 - val_loss: 0.4990 - val_recall: 0.8600
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5147 - recall: 0.7293 - val_loss: 0.5106 - val_recall: 0.8550
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5174 - recall: 0.7386 - val_loss: 0.5075 - val_recall: 0.8698
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5149 - recall: 0.7374 - val_loss: 0.5000 - val_recall: 0.8649
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5223 - recall: 0.7384 - val_loss: 0.4766 - val_recall: 0.7174
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5195 - recall: 0.7322 - val_loss: 0.5064 - val_recall: 0.8870
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5178 - recall: 0.7369 - val_loss: 0.4984 - val_recall: 0.8845
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5145 - recall: 0.7388 - val_loss: 0.4840 - val_recall: 0.7273
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5121 - recall: 0.7353 - val_loss: 0.4975 - val_recall: 0.8673
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5217 - recall: 0.7424 - val_loss: 0.4545 - val_recall: 0.7666
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5048 - recall: 0.7403 - val_loss: 0.4930 - val_recall: 0.7936
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5184 - recall: 0.7034 - val_loss: 0.4640 - val_recall: 0.8231
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5177 - recall: 0.7348 - val_loss: 0.4925 - val_recall: 0.8329
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5206 - recall: 0.7208 - val_loss: 0.4611 - val_recall: 0.6609
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5094 - recall: 0.7168 - val_loss: 0.4855 - val_recall: 0.8575
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5186 - recall: 0.7289 - val_loss: 0.5084 - val_recall: 0.8403
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5087 - recall: 0.7353 - val_loss: 0.4948 - val_recall: 0.8428
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5182 - recall: 0.7438 - val_loss: 0.4959 - val_recall: 0.8354
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5255 - recall: 0.7391 - val_loss: 0.5204 - val_recall: 0.8575
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5213 - recall: 0.7382 - val_loss: 0.4954 - val_recall: 0.8796
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5205 - recall: 0.7433 - val_loss: 0.5035 - val_recall: 0.8845
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5049 - recall: 0.7420 - val_loss: 0.4801 - val_recall: 0.6781

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history01.history['loss'],adam_dropout_smote_history01.history['val_loss'])

• The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history01.history['recall'],adam_dropout_smote_history01.history['val_recall'])

y_train_pred = adam_dropout_smote_model01.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model01.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 546us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.748796
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01), dtype: float64
recall    0.678133
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.77      0.83      0.80      4777
           1       0.82      0.75      0.78      4777

    accuracy                           0.79      9554
   macro avg       0.79      0.79      0.79      9554
weighted avg       0.79      0.79      0.79      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.91      0.82      0.86      1593
           1       0.49      0.68      0.57       407

    accuracy                           0.79      2000
   macro avg       0.70      0.75      0.72      2000
weighted avg       0.82      0.79      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• That came out worse, lets try going back to adjusting on the thousandths.

Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)"

#Initializing the model
adam_dropout_smote_model = Sequential()
adam_dropout_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model.add(Dropout(0.8))
adam_dropout_smote_model.add(Dense(32,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.4))
adam_dropout_smote_model.add(Dense(16,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.2))
adam_dropout_smote_model.add(Dense(8,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.1))
adam_dropout_smote_model.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.003)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history = adam_dropout_smote_model.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.6833 - recall: 0.5378 - val_loss: 0.6247 - val_recall: 0.7273
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6334 - recall: 0.6912 - val_loss: 0.5943 - val_recall: 0.7297
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6053 - recall: 0.7153 - val_loss: 0.5781 - val_recall: 0.7420
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5816 - recall: 0.7239 - val_loss: 0.5674 - val_recall: 0.7371
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5683 - recall: 0.7251 - val_loss: 0.5552 - val_recall: 0.7543
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5585 - recall: 0.7175 - val_loss: 0.5371 - val_recall: 0.7641
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5524 - recall: 0.7342 - val_loss: 0.5581 - val_recall: 0.8108
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5535 - recall: 0.7260 - val_loss: 0.5361 - val_recall: 0.7887
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5294 - recall: 0.7201 - val_loss: 0.5310 - val_recall: 0.7887
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5331 - recall: 0.7147 - val_loss: 0.5384 - val_recall: 0.8329
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5240 - recall: 0.7486 - val_loss: 0.5470 - val_recall: 0.8354
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5221 - recall: 0.7518 - val_loss: 0.5166 - val_recall: 0.8133
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5150 - recall: 0.7446 - val_loss: 0.5214 - val_recall: 0.8157
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5186 - recall: 0.7391 - val_loss: 0.5223 - val_recall: 0.8182
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5176 - recall: 0.7462 - val_loss: 0.5118 - val_recall: 0.8182
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5203 - recall: 0.7498 - val_loss: 0.5106 - val_recall: 0.8182
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5098 - recall: 0.7542 - val_loss: 0.4984 - val_recall: 0.8280
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5124 - recall: 0.7467 - val_loss: 0.4853 - val_recall: 0.8010
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5130 - recall: 0.7382 - val_loss: 0.5020 - val_recall: 0.8206
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5049 - recall: 0.7645 - val_loss: 0.5144 - val_recall: 0.8231
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5018 - recall: 0.7590 - val_loss: 0.4982 - val_recall: 0.8182
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5038 - recall: 0.7436 - val_loss: 0.4950 - val_recall: 0.8403
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5035 - recall: 0.7754 - val_loss: 0.5201 - val_recall: 0.8550
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5027 - recall: 0.7652 - val_loss: 0.4998 - val_recall: 0.8403
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5078 - recall: 0.7588 - val_loss: 0.5085 - val_recall: 0.8600
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5024 - recall: 0.7637 - val_loss: 0.5181 - val_recall: 0.8600
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4987 - recall: 0.7631 - val_loss: 0.5161 - val_recall: 0.8354
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5030 - recall: 0.7461 - val_loss: 0.4991 - val_recall: 0.8256
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5038 - recall: 0.7578 - val_loss: 0.4793 - val_recall: 0.7617
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4999 - recall: 0.7659 - val_loss: 0.4939 - val_recall: 0.7764
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4985 - recall: 0.7629 - val_loss: 0.4921 - val_recall: 0.7813
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4950 - recall: 0.7712 - val_loss: 0.4744 - val_recall: 0.7224
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4972 - recall: 0.7581 - val_loss: 0.4839 - val_recall: 0.7248
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5002 - recall: 0.7354 - val_loss: 0.4717 - val_recall: 0.7248
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5019 - recall: 0.7348 - val_loss: 0.5001 - val_recall: 0.7985
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4976 - recall: 0.7463 - val_loss: 0.4871 - val_recall: 0.7518
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4934 - recall: 0.7464 - val_loss: 0.4988 - val_recall: 0.7641
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4954 - recall: 0.7546 - val_loss: 0.4830 - val_recall: 0.7273
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4909 - recall: 0.7411 - val_loss: 0.4940 - val_recall: 0.7371
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4965 - recall: 0.7507 - val_loss: 0.4887 - val_recall: 0.7641
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7386 - val_loss: 0.4767 - val_recall: 0.7027
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4989 - recall: 0.7138 - val_loss: 0.4902 - val_recall: 0.7224
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4910 - recall: 0.7375 - val_loss: 0.4813 - val_recall: 0.7273
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7446 - val_loss: 0.4853 - val_recall: 0.7715
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4903 - recall: 0.7446 - val_loss: 0.4891 - val_recall: 0.7445
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4901 - recall: 0.7399 - val_loss: 0.4702 - val_recall: 0.7076
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4982 - recall: 0.7241 - val_loss: 0.4849 - val_recall: 0.7396
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4950 - recall: 0.7329 - val_loss: 0.4865 - val_recall: 0.7125
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4962 - recall: 0.7098 - val_loss: 0.4653 - val_recall: 0.8010
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.4854 - recall: 0.7496 - val_loss: 0.4865 - val_recall: 0.7961

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history.history['loss'],adam_dropout_smote_history.history['val_loss'])

• The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets though much more extreme in validation.

#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history.history['recall'],adam_dropout_smote_history.history['val_recall'])

y_train_pred = adam_dropout_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 564us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.83609
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003), dtype: float64
recall    0.796069
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.82      0.75      0.78      4777
           1       0.77      0.84      0.80      4777

    accuracy                           0.79      9554
   macro avg       0.80      0.79      0.79      9554
weighted avg       0.80      0.79      0.79      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.94      0.75      0.83      1593
           1       0.45      0.80      0.57       407

    accuracy                           0.76      2000
   macro avg       0.69      0.77      0.70      2000
weighted avg       0.84      0.76      0.78      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Ok, that moved the bar pretty good. 84% on training and 80 on validation is pretty good.
• Lets see if adjusting batch size helps improve things.

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout, Adjusting LR and Batch size

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32)"

#Initializing the model
adam_dropout_smote_model2 = Sequential()
adam_dropout_smote_model2.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2.add(Dropout(0.8))
adam_dropout_smote_model2.add(Dense(32,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.4))
adam_dropout_smote_model2.add(Dense(16,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.2))
adam_dropout_smote_model2.add(Dense(8,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.1))
adam_dropout_smote_model2.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model2.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history2 = adam_dropout_smote_model2.fit(
    X_train_smote,y_train_smote,
    batch_size=32, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.6685 - recall: 0.5311 - val_loss: 0.5719 - val_recall: 0.7346
Epoch 2/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6080 - recall: 0.7243 - val_loss: 0.5628 - val_recall: 0.7322
Epoch 3/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5825 - recall: 0.6946 - val_loss: 0.5659 - val_recall: 0.7248
Epoch 4/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5829 - recall: 0.7207 - val_loss: 0.5347 - val_recall: 0.7052
Epoch 5/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5738 - recall: 0.7017 - val_loss: 0.5653 - val_recall: 0.7297
Epoch 6/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5572 - recall: 0.7008 - val_loss: 0.5469 - val_recall: 0.8084
Epoch 7/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5646 - recall: 0.7199 - val_loss: 0.5105 - val_recall: 0.7838
Epoch 8/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5562 - recall: 0.7194 - val_loss: 0.5553 - val_recall: 0.7690
Epoch 9/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5507 - recall: 0.7286 - val_loss: 0.5179 - val_recall: 0.8305
Epoch 10/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5487 - recall: 0.7397 - val_loss: 0.5314 - val_recall: 0.8280
Epoch 11/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5452 - recall: 0.7281 - val_loss: 0.5365 - val_recall: 0.8550
Epoch 12/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5520 - recall: 0.7302 - val_loss: 0.5733 - val_recall: 0.8600
Epoch 13/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5476 - recall: 0.7354 - val_loss: 0.5127 - val_recall: 0.8501
Epoch 14/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5470 - recall: 0.7262 - val_loss: 0.5255 - val_recall: 0.8329
Epoch 15/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5364 - recall: 0.7385 - val_loss: 0.5772 - val_recall: 0.8256
Epoch 16/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5401 - recall: 0.7251 - val_loss: 0.5301 - val_recall: 0.8256
Epoch 17/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5404 - recall: 0.7420 - val_loss: 0.5593 - val_recall: 0.8624
Epoch 18/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5412 - recall: 0.7246 - val_loss: 0.5316 - val_recall: 0.8280
Epoch 19/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5264 - recall: 0.7239 - val_loss: 0.5205 - val_recall: 0.7813
Epoch 20/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5313 - recall: 0.7253 - val_loss: 0.5273 - val_recall: 0.8305
Epoch 21/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5441 - recall: 0.7189 - val_loss: 0.5437 - val_recall: 0.8378
Epoch 22/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5417 - recall: 0.7300 - val_loss: 0.5310 - val_recall: 0.8206
Epoch 23/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5353 - recall: 0.7455 - val_loss: 0.5285 - val_recall: 0.8477
Epoch 24/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5328 - recall: 0.7456 - val_loss: 0.5232 - val_recall: 0.8280
Epoch 25/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5410 - recall: 0.7349 - val_loss: 0.4805 - val_recall: 0.8206
Epoch 26/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5309 - recall: 0.7461 - val_loss: 0.5054 - val_recall: 0.8157
Epoch 27/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5391 - recall: 0.7507 - val_loss: 0.5353 - val_recall: 0.8354
Epoch 28/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5287 - recall: 0.7307 - val_loss: 0.5272 - val_recall: 0.8378
Epoch 29/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5369 - recall: 0.7214 - val_loss: 0.4839 - val_recall: 0.8010
Epoch 30/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5273 - recall: 0.7123 - val_loss: 0.5522 - val_recall: 0.8059
Epoch 31/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5332 - recall: 0.6987 - val_loss: 0.5560 - val_recall: 0.8059
Epoch 32/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5286 - recall: 0.7385 - val_loss: 0.5449 - val_recall: 0.7740
Epoch 33/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5337 - recall: 0.7113 - val_loss: 0.4915 - val_recall: 0.7740
Epoch 34/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5424 - recall: 0.7431 - val_loss: 0.5259 - val_recall: 0.8477
Epoch 35/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5331 - recall: 0.7499 - val_loss: 0.5055 - val_recall: 0.8133
Epoch 36/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5244 - recall: 0.7161 - val_loss: 0.5418 - val_recall: 0.8280
Epoch 37/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5326 - recall: 0.7395 - val_loss: 0.5211 - val_recall: 0.8256
Epoch 38/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5295 - recall: 0.7285 - val_loss: 0.5156 - val_recall: 0.8157
Epoch 39/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5317 - recall: 0.7277 - val_loss: 0.5169 - val_recall: 0.8231
Epoch 40/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5266 - recall: 0.7195 - val_loss: 0.5070 - val_recall: 0.8452
Epoch 41/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5246 - recall: 0.7496 - val_loss: 0.5370 - val_recall: 0.8354
Epoch 42/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5295 - recall: 0.7173 - val_loss: 0.5322 - val_recall: 0.8403
Epoch 43/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5383 - recall: 0.7372 - val_loss: 0.5143 - val_recall: 0.8133
Epoch 44/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5254 - recall: 0.7311 - val_loss: 0.5291 - val_recall: 0.8378
Epoch 45/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5266 - recall: 0.7398 - val_loss: 0.5418 - val_recall: 0.8305
Epoch 46/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5295 - recall: 0.7382 - val_loss: 0.5059 - val_recall: 0.8157
Epoch 47/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5273 - recall: 0.7067 - val_loss: 0.5265 - val_recall: 0.8305
Epoch 48/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5302 - recall: 0.7268 - val_loss: 0.5078 - val_recall: 0.8305
Epoch 49/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5420 - recall: 0.7211 - val_loss: 0.4950 - val_recall: 0.8010
Epoch 50/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5367 - recall: 0.7291 - val_loss: 0.5010 - val_recall: 0.8305

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2.history['loss'],adam_dropout_smote_history2.history['val_loss'])

• The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation.
• We can see right around 22 or 23 the train and validation set are very similar and close. We should try exiting around that number and see how it improves things.

y_train_pred = adam_dropout_smote_model2.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 663us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model2.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 552us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.868327
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32), dtype: float64
recall    0.830467
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.84      0.70      0.76      4777
           1       0.74      0.87      0.80      4777

    accuracy                           0.78      9554
   macro avg       0.79      0.78      0.78      9554
weighted avg       0.79      0.78      0.78      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.94      0.69      0.80      1593
           1       0.41      0.83      0.55       407

    accuracy                           0.72      2000
   macro avg       0.67      0.76      0.67      2000
weighted avg       0.83      0.72      0.75      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• That showed some further improvement up to 87/83.
• Lets see if adjusting the epochs will help.

Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23"

#Initializing the model
adam_dropout_smote_model2c = Sequential()
adam_dropout_smote_model2c.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2c.add(Dropout(0.8))
adam_dropout_smote_model2c.add(Dense(32,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.4))
adam_dropout_smote_model2c.add(Dense(16,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.2))
adam_dropout_smote_model2c.add(Dense(8,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.1))
adam_dropout_smote_model2c.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2c.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model2c.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history2c = adam_dropout_smote_model2c.fit(
    X_train_smote,y_train_smote,
    batch_size=32, 
    epochs=23,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.6685 - recall: 0.5311 - val_loss: 0.5719 - val_recall: 0.7346
Epoch 2/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6080 - recall: 0.7243 - val_loss: 0.5628 - val_recall: 0.7322
Epoch 3/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5825 - recall: 0.6946 - val_loss: 0.5659 - val_recall: 0.7248
Epoch 4/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5829 - recall: 0.7207 - val_loss: 0.5347 - val_recall: 0.7052
Epoch 5/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5738 - recall: 0.7017 - val_loss: 0.5653 - val_recall: 0.7297
Epoch 6/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5572 - recall: 0.7008 - val_loss: 0.5469 - val_recall: 0.8084
Epoch 7/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5646 - recall: 0.7199 - val_loss: 0.5105 - val_recall: 0.7838
Epoch 8/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5562 - recall: 0.7194 - val_loss: 0.5553 - val_recall: 0.7690
Epoch 9/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5507 - recall: 0.7286 - val_loss: 0.5179 - val_recall: 0.8305
Epoch 10/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5487 - recall: 0.7397 - val_loss: 0.5314 - val_recall: 0.8280
Epoch 11/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5452 - recall: 0.7281 - val_loss: 0.5365 - val_recall: 0.8550
Epoch 12/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5520 - recall: 0.7302 - val_loss: 0.5733 - val_recall: 0.8600
Epoch 13/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5476 - recall: 0.7354 - val_loss: 0.5127 - val_recall: 0.8501
Epoch 14/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5470 - recall: 0.7262 - val_loss: 0.5255 - val_recall: 0.8329
Epoch 15/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5364 - recall: 0.7385 - val_loss: 0.5772 - val_recall: 0.8256
Epoch 16/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5401 - recall: 0.7251 - val_loss: 0.5301 - val_recall: 0.8256
Epoch 17/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5404 - recall: 0.7420 - val_loss: 0.5593 - val_recall: 0.8624
Epoch 18/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5412 - recall: 0.7246 - val_loss: 0.5316 - val_recall: 0.8280
Epoch 19/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5264 - recall: 0.7239 - val_loss: 0.5205 - val_recall: 0.7813
Epoch 20/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5313 - recall: 0.7253 - val_loss: 0.5273 - val_recall: 0.8305
Epoch 21/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5441 - recall: 0.7189 - val_loss: 0.5437 - val_recall: 0.8378
Epoch 22/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5417 - recall: 0.7300 - val_loss: 0.5310 - val_recall: 0.8206
Epoch 23/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5353 - recall: 0.7455 - val_loss: 0.5285 - val_recall: 0.8477

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2c.history['loss'],adam_dropout_smote_history2c.history['val_loss'])

• The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation. Convergence is close in the final epochs.

y_train_pred = adam_dropout_smote_model2c.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step

array([[ True],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model2c.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 544us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.864559
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23, dtype: float64
recall    0.847666
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23, dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.84      0.69      0.76      4777
           1       0.74      0.86      0.80      4777

    accuracy                           0.78      9554
   macro avg       0.79      0.78      0.78      9554
weighted avg       0.79      0.78      0.78      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.95      0.69      0.80      1593
           1       0.41      0.85      0.55       407

    accuracy                           0.72      2000
   macro avg       0.68      0.77      0.67      2000
weighted avg       0.84      0.72      0.75      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Observations

• Training set dropped a little bit, but validation came up by almost 2 full points.
• Train and validation sets also have the least difference of any other model.
• This will be our final model.

Model Performance Comparison and Final Model Selection

print("Training performance comparison")
training_metrics

Training performance comparison

	recall
Neural Network (64,32,16,8) with SGD	0.429272
Neural Network (64,32,16,8) with SGD + Momentum (0.9)	0.518397
Neural Network (64,32,16,8) with Adam	0.651676
Neural Network (256,128,64,32,16,8) with Adam	0.806214
Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)	0.876533
Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)	0.510221
Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)	0.420278
Neural Network (64,32) with Adam & Dropout (0.5, 0.5)	0.348324
Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)	0.376124
Neural Network (64,32,16,8) with Adam & Batch Normalization	0.683565
Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)	0.413737
Neural Network (64,32,16,8) with SMOTE & SGD	0.805526
Neural Network (64,32,16,8) with SMOTE & Adam	0.941386
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1)	0.793804
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)	0.793804
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)	0.789617
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)	0.748796
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)	0.836090
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32)	0.868327
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23	0.864559

print("Validation set performance comparison")
validation_metrics

Validation set performance comparison

	recall
Neural Network (64,32,16,8) with SGD	0.398034
Neural Network (64,32,16,8) with SGD + Momentum (0.9)	0.405405
Neural Network (64,32,16,8) with Adam	0.479115
Neural Network (256,128,64,32,16,8) with Adam	0.491400
Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)	0.457002
Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)	0.474201
Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)	0.383292
Neural Network (64,32) with Adam & Dropout (0.5, 0.5)	0.304668
Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)	0.331695
Neural Network (64,32,16,8) with Adam & Batch Normalization	0.461916
Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)	0.366093
Neural Network (64,32,16,8) with SMOTE & SGD	0.727273
Neural Network (64,32,16,8) with SMOTE & Adam	0.651106
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1)	0.729730
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)	0.729730
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)	0.724816
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)	0.678133
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)	0.796069
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32)	0.830467
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23	0.847666

# Compare diffs
training_metrics - validation_metrics

	recall
Neural Network (64,32,16,8) with SGD	0.031238
Neural Network (64,32,16,8) with SGD + Momentum (0.9)	0.112992
Neural Network (64,32,16,8) with Adam	0.172561
Neural Network (256,128,64,32,16,8) with Adam	0.314814
Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)	0.419531
Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)	0.036019
Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)	0.036986
Neural Network (64,32) with Adam & Dropout (0.5, 0.5)	0.043655
Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)	0.044429
Neural Network (64,32,16,8) with Adam & Batch Normalization	0.221649
Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)	0.047643
Neural Network (64,32,16,8) with SMOTE & SGD	0.078254
Neural Network (64,32,16,8) with SMOTE & Adam	0.290280
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1)	0.064074
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)	0.064074
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)	0.064801
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)	0.070664
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)	0.040021
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32)	0.037861
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23	0.016893

Observations

Training data

• Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) at 86.8%
• Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23 at 86.4%

Validation data

• While epochs 50 version performs better in training, the epochs 23 version beats it in validation set at 84.7% to 83%.
• Epochs 23 version also is much closer in the 2 values at 1.7% difference vs 3.8% difference.

Conclusion

• For this reason, the Epochs 23 version is our final model selection.

Final Model

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model = "Final Model: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23"

#Initializing the model
final_test_model = Sequential()
final_test_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
final_test_model.add(Dropout(0.8))
final_test_model.add(Dense(32,activation='relu'))
final_test_model.add(Dropout(0.4))
final_test_model.add(Dense(16,activation='relu'))
final_test_model.add(Dropout(0.2))
final_test_model.add(Dense(8,activation='relu'))
final_test_model.add(Dropout(0.1))
final_test_model.add(Dense(1, activation = 'sigmoid'))

#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
final_test_model.compile(loss='binary_crossentropy',optimizer=optimizer)

final_test_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

final_test_history = final_test_model.fit(
    X_train_smote,y_train_smote,
    batch_size=32, 
    epochs=23,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.6685 - val_loss: 0.5719
Epoch 2/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6080 - val_loss: 0.5628
Epoch 3/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5825 - val_loss: 0.5659
Epoch 4/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5829 - val_loss: 0.5347
Epoch 5/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5738 - val_loss: 0.5653
Epoch 6/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5572 - val_loss: 0.5469
Epoch 7/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5646 - val_loss: 0.5105
Epoch 8/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5562 - val_loss: 0.5553
Epoch 9/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5507 - val_loss: 0.5179
Epoch 10/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5487 - val_loss: 0.5314
Epoch 11/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5452 - val_loss: 0.5365
Epoch 12/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5520 - val_loss: 0.5733
Epoch 13/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5476 - val_loss: 0.5127
Epoch 14/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5470 - val_loss: 0.5255
Epoch 15/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5364 - val_loss: 0.5772
Epoch 16/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5401 - val_loss: 0.5301
Epoch 17/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5404 - val_loss: 0.5593
Epoch 18/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5412 - val_loss: 0.5316
Epoch 19/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5264 - val_loss: 0.5205
Epoch 20/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5313 - val_loss: 0.5273
Epoch 21/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5441 - val_loss: 0.5437
Epoch 22/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5417 - val_loss: 0.5310
Epoch 23/23
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5353 - val_loss: 0.5285

Loss function

Classification report

y_test_pred = final_test_model.predict(X_test)
y_test_pred = (y_test_pred > 0.5)
print(y_test_pred)

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
[[False]
 [False]
 [False]
 ...
 [ True]
 [False]
 [False]]

#lets print classification report
cr=classification_report(y_test,y_test_pred)
print(cr)

              precision    recall  f1-score   support

           0       0.94      0.67      0.78      1593
           1       0.39      0.84      0.53       407

    accuracy                           0.70      2000
   macro avg       0.67      0.75      0.66      2000
weighted avg       0.83      0.70      0.73      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_test,y_test_pred)

Actionable Insights and Business Recommendations

Observations

• The majority of model gave recall between 50% to 80% with a few showing low recall in the 30-40% range.
• The data set was imbalanced, and had we had a better data set we might have improved performance even more.
• By using oversampling we were able to improve the models significantly, but there was a lot of noise present in those models.

Conclusion

• The best performing model is "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23", where we used oversampling, a learning rate of 0.01 with the Adam optimizer, used aggressive dropout on each layer, and limited the epochs to 23. While very noisy, This model gave us a recall of 83.5% on the test data set and a difference of 1.7% between the Training and Validation sets.
• By using this model, the bank can identify customers who are willing to leave the bank and enact an action plan to retain these customers.

Business Recommendations

In our analysis, we found that customers who have less than 1 year or more than 9 of Tenure are more likely to churn. These could be newer customers lured in by a promotion recently and have no reason to remain loyal, or on the other end, customers who are now takign advantage of their credit history to move to a different card that offers some incentives. Some effort should be invested to try and improve these customers loyalty. Perhaps by offering lower interest rates (or promotional rates), or some other incentives, these customers would remain.

Customers with no Balance or a Balance between 100-150K are also more likely to churn. For the higher Balance customers this suggests that they may be outgrowing the need for credit cards or finding alternate solutions. The Bank should look for ways to engage these customers with targeted marketing. Those with $0 Balance likely only have a credit card account (backed up by the majority of customers having a single NumOfProducts) and so are not particularly tied to the bank. The Bank should focus effort on getting these customers into additional product relationships or offer promotional incentives to place money into an account.

Finally, customers who are inactive are more likely to churn, possibly indicating dissatisfaction or indifference, leading to closing of the account. The Bank should encourage these customers to utilize their card or other bank services more often to keep customers like this from departing.

print('\n'*10)

Other models tried

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout and LR (0.01)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01)"

#Initializing the model
adam_dropout_smote_model3 = Sequential()
adam_dropout_smote_model3.add(Dense(128,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model3.add(Dropout(0.8))
adam_dropout_smote_model3.add(Dense(64,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.4))
adam_dropout_smote_model3.add(Dense(32,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.2))
adam_dropout_smote_model3.add(Dense(16,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.2))
adam_dropout_smote_model3.add(Dense(8,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.1))
adam_dropout_smote_model3.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
metric = 'accuracy'
# metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

#  compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model3.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 128)            │         1,536 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 128)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 64)             │         8,256 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_4 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_5 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 12,545 (49.00 KB)

 Trainable params: 12,545 (49.00 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history3 = adam_dropout_smote_model3.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - accuracy: 0.5878 - loss: 0.6655 - val_accuracy: 0.7315 - val_loss: 0.5162
Epoch 2/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7017 - loss: 0.5857 - val_accuracy: 0.7595 - val_loss: 0.5130
Epoch 3/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7089 - loss: 0.5835 - val_accuracy: 0.7405 - val_loss: 0.4941
Epoch 4/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7245 - loss: 0.5664 - val_accuracy: 0.7290 - val_loss: 0.4787
Epoch 5/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7235 - loss: 0.5569 - val_accuracy: 0.7200 - val_loss: 0.5031
Epoch 6/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7263 - loss: 0.5517 - val_accuracy: 0.7145 - val_loss: 0.4965
Epoch 7/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7404 - loss: 0.5383 - val_accuracy: 0.7285 - val_loss: 0.4749
Epoch 8/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7472 - loss: 0.5249 - val_accuracy: 0.7470 - val_loss: 0.4711
Epoch 9/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7461 - loss: 0.5336 - val_accuracy: 0.7255 - val_loss: 0.4975
Epoch 10/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7406 - loss: 0.5324 - val_accuracy: 0.7445 - val_loss: 0.4866
Epoch 11/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7417 - loss: 0.5329 - val_accuracy: 0.7530 - val_loss: 0.4776
Epoch 12/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7484 - loss: 0.5295 - val_accuracy: 0.7685 - val_loss: 0.4467
Epoch 13/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7463 - loss: 0.5338 - val_accuracy: 0.7310 - val_loss: 0.4677
Epoch 14/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7534 - loss: 0.5229 - val_accuracy: 0.7605 - val_loss: 0.4698
Epoch 15/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7474 - loss: 0.5373 - val_accuracy: 0.7390 - val_loss: 0.4886
Epoch 16/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7515 - loss: 0.5269 - val_accuracy: 0.7810 - val_loss: 0.4694
Epoch 17/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7524 - loss: 0.5244 - val_accuracy: 0.7050 - val_loss: 0.5172
Epoch 18/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7517 - loss: 0.5315 - val_accuracy: 0.7545 - val_loss: 0.4835
Epoch 19/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7517 - loss: 0.5261 - val_accuracy: 0.7585 - val_loss: 0.4726
Epoch 20/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7482 - loss: 0.5296 - val_accuracy: 0.7500 - val_loss: 0.5085
Epoch 21/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7527 - loss: 0.5224 - val_accuracy: 0.7480 - val_loss: 0.4813
Epoch 22/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7471 - loss: 0.5269 - val_accuracy: 0.7570 - val_loss: 0.4704
Epoch 23/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7495 - loss: 0.5265 - val_accuracy: 0.7920 - val_loss: 0.4641
Epoch 24/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7516 - loss: 0.5222 - val_accuracy: 0.7610 - val_loss: 0.4654
Epoch 25/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7484 - loss: 0.5225 - val_accuracy: 0.7715 - val_loss: 0.4747
Epoch 26/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7496 - loss: 0.5261 - val_accuracy: 0.7560 - val_loss: 0.4697
Epoch 27/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7462 - loss: 0.5256 - val_accuracy: 0.7890 - val_loss: 0.4576
Epoch 28/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7504 - loss: 0.5201 - val_accuracy: 0.7740 - val_loss: 0.4672
Epoch 29/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7580 - loss: 0.5120 - val_accuracy: 0.7460 - val_loss: 0.4816
Epoch 30/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7515 - loss: 0.5303 - val_accuracy: 0.7860 - val_loss: 0.4779
Epoch 31/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7500 - loss: 0.5186 - val_accuracy: 0.7605 - val_loss: 0.4611
Epoch 32/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7505 - loss: 0.5242 - val_accuracy: 0.7590 - val_loss: 0.4871
Epoch 33/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7528 - loss: 0.5291 - val_accuracy: 0.7725 - val_loss: 0.4795
Epoch 34/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7663 - loss: 0.5094 - val_accuracy: 0.7675 - val_loss: 0.4723
Epoch 35/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7540 - loss: 0.5148 - val_accuracy: 0.7840 - val_loss: 0.4772
Epoch 36/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7529 - loss: 0.5164 - val_accuracy: 0.7945 - val_loss: 0.4607
Epoch 37/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7552 - loss: 0.5207 - val_accuracy: 0.7840 - val_loss: 0.4550
Epoch 38/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7571 - loss: 0.5123 - val_accuracy: 0.7845 - val_loss: 0.4601
Epoch 39/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7600 - loss: 0.5150 - val_accuracy: 0.7585 - val_loss: 0.4599
Epoch 40/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7572 - loss: 0.5129 - val_accuracy: 0.7535 - val_loss: 0.5054
Epoch 41/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7488 - loss: 0.5352 - val_accuracy: 0.7655 - val_loss: 0.4940
Epoch 42/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7515 - loss: 0.5170 - val_accuracy: 0.7600 - val_loss: 0.4683
Epoch 43/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7510 - loss: 0.5168 - val_accuracy: 0.7870 - val_loss: 0.4544
Epoch 44/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7593 - loss: 0.5134 - val_accuracy: 0.7725 - val_loss: 0.4579
Epoch 45/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7530 - loss: 0.5202 - val_accuracy: 0.7835 - val_loss: 0.4668
Epoch 46/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7619 - loss: 0.5089 - val_accuracy: 0.7550 - val_loss: 0.4740
Epoch 47/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7507 - loss: 0.5162 - val_accuracy: 0.7545 - val_loss: 0.4833
Epoch 48/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7547 - loss: 0.5074 - val_accuracy: 0.7715 - val_loss: 0.4797
Epoch 49/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7523 - loss: 0.5227 - val_accuracy: 0.7995 - val_loss: 0.4472
Epoch 50/50
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7517 - loss: 0.5260 - val_accuracy: 0.7820 - val_loss: 0.4573

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history3.history['loss'],adam_dropout_smote_history3.history['val_loss'])

y_train_pred = adam_dropout_smote_model3.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 744us/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model3.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 596us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.779987
Name: Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01), dtype: float64
recall    0.737101
Name: Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.78      0.80      0.79      4777
           1       0.80      0.78      0.79      4777

    accuracy                           0.79      9554
   macro avg       0.79      0.79      0.79      9554
weighted avg       0.79      0.79      0.79      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.92      0.79      0.85      1593
           1       0.48      0.74      0.58       407

    accuracy                           0.78      2000
   macro avg       0.70      0.77      0.72      2000
weighted avg       0.83      0.78      0.80      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003)

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

batch_size_opt= 32
epochs_opt = 50

#Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2)
print(X_train_smote.shape[1])

11

model_name = "Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003)"

#Initializing the model
adam_dropout_smote_model4 = Sequential()
adam_dropout_smote_model4.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model4.add(Dropout(0.8))
adam_dropout_smote_model4.add(Dense(32,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.4))
adam_dropout_smote_model4.add(Dense(16,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.2))
adam_dropout_smote_model4.add(Dense(8,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.2))
adam_dropout_smote_model4.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.003)

# uncomment one of the following lines to define the metric to be used
metric = 'accuracy'
# metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model4.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model4.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history4 = adam_dropout_smote_model4.fit(
    X_train_smote,y_train_smote,
    batch_size=batch_size_opt, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - accuracy: 0.5472 - loss: 0.6852 - val_accuracy: 0.7180 - val_loss: 0.5740
Epoch 2/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.6804 - loss: 0.6245 - val_accuracy: 0.7025 - val_loss: 0.5768
Epoch 3/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.6957 - loss: 0.5968 - val_accuracy: 0.7140 - val_loss: 0.5534
Epoch 4/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7146 - loss: 0.5799 - val_accuracy: 0.7125 - val_loss: 0.5584
Epoch 5/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7206 - loss: 0.5689 - val_accuracy: 0.7170 - val_loss: 0.5222
Epoch 6/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7231 - loss: 0.5554 - val_accuracy: 0.7165 - val_loss: 0.5237
Epoch 7/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7407 - loss: 0.5478 - val_accuracy: 0.7310 - val_loss: 0.5103
Epoch 8/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7372 - loss: 0.5396 - val_accuracy: 0.7340 - val_loss: 0.5214
Epoch 9/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7473 - loss: 0.5305 - val_accuracy: 0.7300 - val_loss: 0.5289
Epoch 10/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7471 - loss: 0.5290 - val_accuracy: 0.7390 - val_loss: 0.5182
Epoch 11/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7486 - loss: 0.5269 - val_accuracy: 0.7435 - val_loss: 0.5032
Epoch 12/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7562 - loss: 0.5243 - val_accuracy: 0.7130 - val_loss: 0.5239
Epoch 13/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7541 - loss: 0.5268 - val_accuracy: 0.7285 - val_loss: 0.5274
Epoch 14/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7537 - loss: 0.5238 - val_accuracy: 0.7460 - val_loss: 0.5089
Epoch 15/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7493 - loss: 0.5198 - val_accuracy: 0.6990 - val_loss: 0.5281
Epoch 16/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7537 - loss: 0.5152 - val_accuracy: 0.7120 - val_loss: 0.5295
Epoch 17/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7523 - loss: 0.5145 - val_accuracy: 0.7400 - val_loss: 0.5180
Epoch 18/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7537 - loss: 0.5157 - val_accuracy: 0.7310 - val_loss: 0.5147
Epoch 19/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7604 - loss: 0.5087 - val_accuracy: 0.7370 - val_loss: 0.5086
Epoch 20/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7529 - loss: 0.5147 - val_accuracy: 0.7760 - val_loss: 0.5041
Epoch 21/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7536 - loss: 0.5139 - val_accuracy: 0.7100 - val_loss: 0.5160
Epoch 22/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7588 - loss: 0.5081 - val_accuracy: 0.7140 - val_loss: 0.4888
Epoch 23/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7519 - loss: 0.5091 - val_accuracy: 0.7090 - val_loss: 0.5244
Epoch 24/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7575 - loss: 0.5139 - val_accuracy: 0.7170 - val_loss: 0.5285
Epoch 25/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7538 - loss: 0.5118 - val_accuracy: 0.7100 - val_loss: 0.5079
Epoch 26/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7597 - loss: 0.5104 - val_accuracy: 0.7025 - val_loss: 0.5467
Epoch 27/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7614 - loss: 0.5065 - val_accuracy: 0.7055 - val_loss: 0.5337
Epoch 28/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7507 - loss: 0.5067 - val_accuracy: 0.7195 - val_loss: 0.5143
Epoch 29/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7521 - loss: 0.5099 - val_accuracy: 0.7830 - val_loss: 0.4965
Epoch 30/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7550 - loss: 0.5001 - val_accuracy: 0.7100 - val_loss: 0.5209
Epoch 31/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7551 - loss: 0.5022 - val_accuracy: 0.7330 - val_loss: 0.5158
Epoch 32/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7571 - loss: 0.5059 - val_accuracy: 0.7145 - val_loss: 0.4988
Epoch 33/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7588 - loss: 0.5047 - val_accuracy: 0.7275 - val_loss: 0.5053
Epoch 34/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7538 - loss: 0.5074 - val_accuracy: 0.7290 - val_loss: 0.4931
Epoch 35/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7620 - loss: 0.5045 - val_accuracy: 0.7780 - val_loss: 0.4965
Epoch 36/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7606 - loss: 0.4981 - val_accuracy: 0.7785 - val_loss: 0.4969
Epoch 37/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7549 - loss: 0.5117 - val_accuracy: 0.7845 - val_loss: 0.5040
Epoch 38/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7620 - loss: 0.4974 - val_accuracy: 0.7145 - val_loss: 0.5134
Epoch 39/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7617 - loss: 0.5004 - val_accuracy: 0.7360 - val_loss: 0.5017
Epoch 40/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7617 - loss: 0.4968 - val_accuracy: 0.7230 - val_loss: 0.4860
Epoch 41/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7561 - loss: 0.5061 - val_accuracy: 0.7910 - val_loss: 0.4901
Epoch 42/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7531 - loss: 0.5032 - val_accuracy: 0.7465 - val_loss: 0.4763
Epoch 43/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7556 - loss: 0.5039 - val_accuracy: 0.7395 - val_loss: 0.5131
Epoch 44/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7570 - loss: 0.5017 - val_accuracy: 0.7195 - val_loss: 0.5359
Epoch 45/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7629 - loss: 0.4977 - val_accuracy: 0.7000 - val_loss: 0.5080
Epoch 46/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7555 - loss: 0.5025 - val_accuracy: 0.7395 - val_loss: 0.4976
Epoch 47/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7567 - loss: 0.4999 - val_accuracy: 0.7755 - val_loss: 0.5085
Epoch 48/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7583 - loss: 0.5013 - val_accuracy: 0.7855 - val_loss: 0.4902
Epoch 49/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7553 - loss: 0.5047 - val_accuracy: 0.7845 - val_loss: 0.4891
Epoch 50/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.7629 - loss: 0.5028 - val_accuracy: 0.8020 - val_loss: 0.4726

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history4.history['loss'],adam_dropout_smote_history4.history['val_loss'])

y_train_pred = adam_dropout_smote_model4.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 704us/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model4.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 544us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.753192
Name: Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003), dtype: float64
recall    0.692875
Name: Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.77      0.84      0.80      4777
           1       0.82      0.75      0.79      4777

    accuracy                           0.80      9554
   macro avg       0.80      0.80      0.80      9554
weighted avg       0.80      0.80      0.80      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.91      0.83      0.87      1593
           1       0.51      0.69      0.59       407

    accuracy                           0.80      2000
   macro avg       0.71      0.76      0.73      2000
weighted avg       0.83      0.80      0.81      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout, Adjusting LR and Batch size

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32)"

#Initializing the model
adam_dropout_smote_model2b = Sequential()
adam_dropout_smote_model2b.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2b.add(Dropout(0.8))
adam_dropout_smote_model2b.add(Dense(32,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.4))
adam_dropout_smote_model2b.add(Dense(16,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.2))
adam_dropout_smote_model2b.add(Dense(8,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.2))
adam_dropout_smote_model2b.add(Dense(1, activation = 'sigmoid'))

# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()

# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2b.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])

adam_dropout_smote_model2b.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 64)             │           768 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 32)             │         2,080 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 32)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 16)             │           528 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 16)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 8)              │           136 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_3 (Dropout)             │ (None, 8)              │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │             9 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,521 (13.75 KB)

 Trainable params: 3,521 (13.75 KB)

 Non-trainable params: 0 (0.00 B)

adam_dropout_smote_history2b = adam_dropout_smote_model2b.fit(
    X_train_smote,y_train_smote,
    batch_size=32, 
    epochs=epochs,
    verbose=1,
    validation_data = (X_val,y_val))

Epoch 1/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.6699 - recall: 0.5114 - val_loss: 0.5946 - val_recall: 0.7666
Epoch 2/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.6169 - recall: 0.7236 - val_loss: 0.5770 - val_recall: 0.7764
Epoch 3/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5914 - recall: 0.7060 - val_loss: 0.5366 - val_recall: 0.7912
Epoch 4/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5888 - recall: 0.7379 - val_loss: 0.5668 - val_recall: 0.8452
Epoch 5/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5746 - recall: 0.7198 - val_loss: 0.5906 - val_recall: 0.8157
Epoch 6/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5612 - recall: 0.7184 - val_loss: 0.5411 - val_recall: 0.8354
Epoch 7/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5615 - recall: 0.7130 - val_loss: 0.5524 - val_recall: 0.8575
Epoch 8/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5483 - recall: 0.7159 - val_loss: 0.5149 - val_recall: 0.8231
Epoch 9/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5520 - recall: 0.7134 - val_loss: 0.5206 - val_recall: 0.8354
Epoch 10/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5465 - recall: 0.7361 - val_loss: 0.4860 - val_recall: 0.7322
Epoch 11/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5398 - recall: 0.7232 - val_loss: 0.5323 - val_recall: 0.7985
Epoch 12/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5501 - recall: 0.7328 - val_loss: 0.5591 - val_recall: 0.8157
Epoch 13/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5431 - recall: 0.7303 - val_loss: 0.5660 - val_recall: 0.8305
Epoch 14/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5482 - recall: 0.7276 - val_loss: 0.5355 - val_recall: 0.8673
Epoch 15/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5363 - recall: 0.7521 - val_loss: 0.5096 - val_recall: 0.8477
Epoch 16/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5393 - recall: 0.7474 - val_loss: 0.5106 - val_recall: 0.8059
Epoch 17/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5374 - recall: 0.7400 - val_loss: 0.5304 - val_recall: 0.8231
Epoch 18/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5407 - recall: 0.7383 - val_loss: 0.5167 - val_recall: 0.8501
Epoch 19/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5299 - recall: 0.7545 - val_loss: 0.5078 - val_recall: 0.8182
Epoch 20/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5370 - recall: 0.7340 - val_loss: 0.4968 - val_recall: 0.7224
Epoch 21/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5402 - recall: 0.7292 - val_loss: 0.5176 - val_recall: 0.8329
Epoch 22/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5257 - recall: 0.7416 - val_loss: 0.4756 - val_recall: 0.7543
Epoch 23/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5391 - recall: 0.7345 - val_loss: 0.5333 - val_recall: 0.8624
Epoch 24/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5376 - recall: 0.7449 - val_loss: 0.5289 - val_recall: 0.8722
Epoch 25/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5484 - recall: 0.7580 - val_loss: 0.4934 - val_recall: 0.7887
Epoch 26/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5411 - recall: 0.7430 - val_loss: 0.5221 - val_recall: 0.8280
Epoch 27/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5449 - recall: 0.7298 - val_loss: 0.4899 - val_recall: 0.7518
Epoch 28/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5452 - recall: 0.7362 - val_loss: 0.5094 - val_recall: 0.8182
Epoch 29/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5320 - recall: 0.7208 - val_loss: 0.5611 - val_recall: 0.8378
Epoch 30/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5331 - recall: 0.7290 - val_loss: 0.4912 - val_recall: 0.7027
Epoch 31/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5316 - recall: 0.7329 - val_loss: 0.4894 - val_recall: 0.8206
Epoch 32/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5408 - recall: 0.7319 - val_loss: 0.5021 - val_recall: 0.8108
Epoch 33/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5383 - recall: 0.7396 - val_loss: 0.4913 - val_recall: 0.7396
Epoch 34/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5340 - recall: 0.7200 - val_loss: 0.5059 - val_recall: 0.7420
Epoch 35/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5355 - recall: 0.7367 - val_loss: 0.5086 - val_recall: 0.7445
Epoch 36/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5324 - recall: 0.7429 - val_loss: 0.5275 - val_recall: 0.8231
Epoch 37/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5291 - recall: 0.7265 - val_loss: 0.5237 - val_recall: 0.7838
Epoch 38/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5234 - recall: 0.7638 - val_loss: 0.5275 - val_recall: 0.8526
Epoch 39/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5306 - recall: 0.7611 - val_loss: 0.4727 - val_recall: 0.8010
Epoch 40/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5344 - recall: 0.7553 - val_loss: 0.4775 - val_recall: 0.7273
Epoch 41/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5410 - recall: 0.6939 - val_loss: 0.4986 - val_recall: 0.8059
Epoch 42/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5335 - recall: 0.7168 - val_loss: 0.4970 - val_recall: 0.8231
Epoch 43/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5330 - recall: 0.7431 - val_loss: 0.4850 - val_recall: 0.7961
Epoch 44/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5290 - recall: 0.7705 - val_loss: 0.5077 - val_recall: 0.8182
Epoch 45/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5226 - recall: 0.7556 - val_loss: 0.4974 - val_recall: 0.8206
Epoch 46/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5263 - recall: 0.7578 - val_loss: 0.5254 - val_recall: 0.8157
Epoch 47/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5274 - recall: 0.7476 - val_loss: 0.5203 - val_recall: 0.8428
Epoch 48/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5366 - recall: 0.7580 - val_loss: 0.4795 - val_recall: 0.7887
Epoch 49/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5366 - recall: 0.7382 - val_loss: 0.4790 - val_recall: 0.7543
Epoch 50/50
299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.5422 - recall: 0.7380 - val_loss: 0.4900 - val_recall: 0.7936

Loss function

#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2b.history['loss'],adam_dropout_smote_history2b.history['val_loss'])

y_train_pred = adam_dropout_smote_model2b.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred

299/299 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step

array([[False],
       [ True],
       [ True],
       ...,
       [ True],
       [False],
       [ True]])

y_val_pred = adam_dropout_smote_model2b.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred

63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 561us/step

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)

print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])

recall    0.840695
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32), dtype: float64
recall    0.793612
Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32), dtype: float64

Classification report

print(classification_report(y_train_smote,y_train_pred))

              precision    recall  f1-score   support

           0       0.82      0.74      0.78      4777
           1       0.76      0.84      0.80      4777

    accuracy                           0.79      9554
   macro avg       0.79      0.79      0.79      9554
weighted avg       0.79      0.79      0.79      9554

print(classification_report(y_val,y_val_pred))

              precision    recall  f1-score   support

           0       0.93      0.71      0.81      1593
           1       0.41      0.79      0.54       407

    accuracy                           0.73      2000
   macro avg       0.67      0.75      0.68      2000
weighted avg       0.83      0.73      0.75      2000

Confusion matrix

#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)

#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)